SciBERT: A Pretrained Language Model for Scientific Text

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3615–3620, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3615 SCIBERT: A Pretrained Language Model for Scientific Text Iz Beltagy Kyle Lo Arman Cohan Allen Institute for Artificial Intelligence, Seattle, WA, USA {beltagy,kylel,armanc}@allenai.org Abstract Obtaining large-scale annotated data for NLP tasks in the scientific domain is challenging and expensive. We release SCIBERT, a pre- trained language model based on BERT (De- vlin et al., 2019) to address the lack of high- quality, large-scale labeled scientific data. SCIBERT leverages unsupervised pretraining on a large multi-domain corpus of scien- tific publications to improve performance on downstream scientific NLP tasks. We eval- uate on a suite of tasks including sequence tagging, sentence classification and depen- dency parsing, with datasets from a variety of scientific domains. We demonstrate sta- tistically significant improvements over BERT and achieve new state-of-the-art results on sev- eral of these tasks. The code and pretrained models are available at https://github. com/allenai/scibert/. 1 Introduction The exponential increase in the volume of scien- tific publications in the past decades has made NLP an essential tool for large-scale knowledge extraction and machine reading of these docu- ments. Recent progress in NLP has been driven by the adoption of deep neural models, but train- ing such models often requires large amounts of labeled data. In general domains, large-scale train- ing data is often possible to obtain through crowd- sourcing, but in scientific domains, annotated data is difficult and expensive to collect due to the ex- pertise required for quality annotation. As shown through ELMo (Peters et al., 2018), GPT (Radford et al., 2018) and BERT (Devlin et al., 2019), unsupervised pretraining of language models on large corpora significantly improves performance on many NLP tasks. These models return contextualized embeddings for each token which can be passed into minimal task-specific neural architectures. Leveraging the success of un- supervised pretraining has become especially im- portant especially when task-specific annotations are difficult to obtain, like in scientific NLP. Yet while both BERT and ELMo have released pre- trained models, they are still trained on general do- main corpora such as news articles and Wikipedia. In this work, we make the following contribu- tions: (i) We release SCIBERT, a new resource demon- strated to improve performance on a range of NLP tasks in the scientific domain. SCIBERT is a pre- trained language model based on BERT but trained on a large corpus of scientific text. (ii) We perform extensive experimentation to investigate the performance of finetuning ver- sus task-specific architectures atop frozen embed- dings, and the effect of having an in-domain vo- cabulary. (iii) We evaluate SCIBERT on a suite of tasks in the scientific domain, and achieve new state-of- the-art (SOTA) results on many of these tasks. 2 Methods Background The BERT model architecture (De- vlin et al., 2019) is based on a multilayer bidirec- tional Transformer (Vaswani et al., 2017). …

Content truncated to 500 words