Scientific Keyphrase Identification and Classification by Pre-Trained Language Models Intermediate Task Transfer Learning

Proceedings of the 28th International Conference on Computational Linguistics, pages 5409–5419 Barcelona, Spain (Online), December 8-13, 2020 5409 Scientific Keyphrase Identification and Classification by Pre-Trained Language Models Intermediate Task Transfer Learning Seo Yeon Park Computer Science Department University of Illinois at Chicago spark313@uic.edu Cornelia Caragea Computer Science Department University of Illinois at Chicago cornelia@uic.edu Abstract Scientific keyphrase identification and classification is the task of detecting and classifying keyphrases from scholarly text with their types from a set of predefined classes. This task has a wide range of benefits, but it is still challenging in performance due to the lack of large amounts of labeled data required for training deep neural models. In order to overcome this challenge, we explore pre-trained language models BERT and SciBERT with intermediate task transfer learn- ing, using 42 data-rich related intermediate-target task combinations. We reveal that intermediate task transfer learning on SciBERT induces a better starting point for target task fine-tuning com- pared with BERT and achieves competitive performance in scientific keyphrase identification and classification compared to both previous works and strong baselines. Interestingly, we observe that BERT with intermediate task transfer learning fails to improve the performance of scientific keyphrase identification and classification potentially due to significant catastrophic forgetting. This result highlights that scientific knowledge achieved during the pre-training of language mod- els on large scientific collections plays an important role in the target tasks. We also observe that sequence tagging related intermediate tasks, especially syntactic structure learning tasks such as POS Tagging, tend to work best for scientific keyphrase identification and classification. 1 Introduction Scientific Keyphrase Identification and Classification (SKIC) is the task of identifying and classifying scientific terms in research papers. An effective keyphrase identification and classification system can benefit a wide range of natural language processing and information retrieval tasks including question answering (Quarteroni and Manandhar, 2006), question generation (Subramanian et al., 2018), and expert finding (Chen et al., 2015; Augenstein et al., 2017). Several works formulate SKIC as a classification task using neural methods and word embeddings (Luan et al., 2018; Augenstein et al., 2017; Liu et al., 2017) and show promising results compared to previous hand-crafted feature processing with sequence labeling such as Conditional Random Fields (Lee et al., 2017). Scientific keyphrases are very diverse and this diversity leads to a burden for the creation of large dataset collections, which results in data scarcity problems for deep neural networks since domain experts are required to obtain reliable annotations for keyphrase identification and classification. In order to overcome the small size data problem of SKIC, Augenstein and Søgaard (2017) propose deep multi-task learning and reveal that several related tasks such as noun and verb phrase chunking, super-sense tagging, and multi-word expression identification are helpful to SKIC. However, the results of this approach are still low (Augenstein et al., 2017), which implies that there is room for improvement. Recent works in natural language processing show that employing unsupervised pre-trained language models such as BERT (Devlin et al., 2019) and SciBERT (Beltagy …

Content truncated to 500 words