New transformer-based model for identifying alloy properties

Determining alloy properties can be expensive and time consuming. Experiments with alloys are often resource intensive. Alloy calculations can also be very complex, with a seemingly infinite number of configurations of properties.

The properties of the alloy can be determined using density functional theory (DFT) calculations. However, this method has limitations and can be very time-consuming, especially for complex alloys. Amir Barati Farimani and his team aim to reduce both the time and cost of this process, and their recent research has led to the development of AlloyBert, a modeling tool designed to predict alloy properties. This led to the creation.

AlloyBert is a transformer-based model that allows researchers to input simple English descriptors and get the desired output. Descriptors can include information such as the temperature at which the alloy was processed and the chemical composition of the alloy. AlloyBert uses this information to predict the modulus or yield strength of the alloy.

Associate Professor of Mechanical Engineering Barati Fairmani and his team specifically designed AlloyBert to reduce both the time and cost typically required to characterize alloys. Most language learning models require users to enter information using very precise wording, which is a time-consuming process. By making AlloyBert a transformer-based model, users have more input flexibility.

“We wanted a model that would allow us to easily capture certain physical properties without worrying too much about what information we had or whether it was in a particular format.” says Akshat Chaudhary, master’s student in materials science and engineering. Importantly, AlloyBert allows for a higher level of flexibility. ”

AlloyBert’s basic model is an existing encoder, RoBERTa. RoBERTa was used due to its self-attention mechanism, a feature that allows the model to determine the importance of certain words in a sentence. This self-attention mechanism was incorporated into the training of AlloyBert using two datasets of alloy properties. AlloyBert has tweaked the RoBERTa model. The results of the study showed that the transformer model can be used as an effective tool to predict the properties of alloys.

AlloyBert currently has two deviations that the team would like to investigate further. The accuracy of AlloyBert’s predictions does not always match the level of detail of the input. The team expected that the more information they provided AlloyBert, the more accurate its output would be.

However, their experiments showed that in some cases, inputting the least amount of data yields the most accurate output. The team speculates that this may be due to AlloyBert’s training being limited to two datasets.

“Training a model on a very large corpus can give you more consistent results,” Chaudhari said.

The second deviation was discovered because the research team employed two training strategies. One method involves first pretraining and then fine-tuning the model, and the other involves only fine-tuning the model. The team hypothesized that a method that uses both pre-training and fine-tuning would yield more accurate output. This occurred 1 in 8 times in each dataset.

Although their hypothesis was largely supported, they found that in some cases, just tweaking the model can yield better results compared to some inputs with more information. The researchers predict that this deviation may be due to the use of masked language models (MLM) in pre-training. Future studies may use alternative pre-training models.

Overall, this research and the development of AlloyBert opens the door to many possibilities. In addition to the two deviations mentioned above, AlloyBert’s code can also be further developed to identify materials other than alloys. Fairmani’s team also envisions developing a model that would do the opposite of AlloyBert: take the properties of an alloy and decompose it into its constituent elements.

In general, transformer-based models have proven to be potentially valuable tools for future scientific research. “Scientific applications require specific and precise answers, and existing research shows that there is ample scope for doing so. These models provide better results than existing methods. You can train in such a way that you get the most out of it,” Chaudhary explains.

The findings have been published on the arXiv preprint server, and AlloyBert’s software is now accessible on GitHub.