MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3621–3639 July 9-14, 2023 ©2023 Association for Computational Linguistics MatSci-NLP: Evaluating Scientific Language Models on Materials Science Language Tasks Using Text-to-Schema Modeling Yu Song1,∗ Santiago Miret2,∗ Bang Liu1,† 1University of Montreal / Mila - Quebec AI, 2Intel Labs {yu.song, bang.liu}@umontreal.ca {santiago.miret}@intel.com Abstract We present MatSci-NLP, a natural language benchmark for evaluating the performance of natural language processing (NLP) models on materials science text. We construct the bench- mark from publicly available materials science text data to encompass seven different NLP tasks, including conventional NLP tasks like named entity recognition and relation classifica- tion, as well as NLP tasks specific to materials science, such as synthesis action retrieval which relates to creating synthesis procedures for ma- terials. We study various BERT-based models pretrained on different scientific text corpora on MatSci-NLP to understand the impact of pre- training strategies on understanding materials science text. Given the scarcity of high-quality annotated data in the materials science domain, we perform our fine-tuning experiments with limited training data to encourage the general- ize across MatSci-NLP tasks. Our experiments in this low-resource training setting show that language models pretrained on scientific text outperform BERT trained on general text. Mat- BERT, a model pretrained specifically on mate- rials science journals, generally performs best for most tasks. Moreover, we propose a uni- fied text-to-schema for multitask learning on MatSci-NLP and compare its performance with traditional fine-tuning methods. In our analy- sis of different training methods, we find that our proposed text-to-schema methods inspired by question-answering consistently outperform single and multitask NLP fine-tuning methods. The code and datasets are publicly available1. 1 Introduction Materials science comprises an interdisciplinary scientific field that studies the behavior, properties and applications of matter that make up materials systems. As such, materials science often requires ∗ Equal contribution. † Corresponding author. Canada CIFAR AI Chair. 1https://github.com/BangLab-UdeM-Mila/ NLP4MatSci-ACL23 deep understanding of a diverse set of scientific dis- ciplines to meaningfully further the state of the art. This interdisciplinary nature, along with the great technological impact of materials advances and growing research work at the intersection of ma- chine learning and materials science (Miret et al.; Pilania, 2021; Choudhary et al., 2022), makes the challenge of developing and evaluating natural lan- guage processing (NLP) models on materials sci- ence text both interesting and exacting. The vast amount of materials science knowledge stored in textual format, such as journal articles, patents and technical reports, creates a tremendous opportunity to develop and build NLP tools to cre- ate and understand advanced materials. These tools could in turn enable faster discovery, synthesis and deployment of new materials into a wide variety of application, including clean energy, sustainable manufacturing and devices. Understanding, processing, and training lan- guage models for scientific text presents distinctive challenges that have given rise to the creation of specialized models and techniques that we review in Section 2. Additionally, evaluating models on scientific language understanding tasks, especially in materials …

Content truncated to 500 words