Intriguing Properties of Compression on Multilingual Models

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 942 - 951 December 7-11, 2022 ©2022 Association for Computational Linguistics Entity Extraction in Low Resource Domains with Selective Pre-training of Large Language Models Aniruddha Mahapatra1+, Sharmila Reddy Nangi2+ Aparna Garimella3, Anandhavelu Natarajan3 1Carnegie Mellon University, USA 2Stanford University, USA 3Adobe Research, India amahapat@andrew.cmu.edu1 srnangi@stanford.edu2 {garimell, anandvn}@adobe.com3 Abstract Transformer-based language models trained on large natural language corpora have been very useful in downstream entity extraction tasks. However, they often result in poor perfor- mances when applied to domains that are differ- ent from those they are pretrained on. Contin- ued pretraining using unlabeled data from tar- get domains can help improve the performances of these language models on the downstream tasks. However, using all of the available unla- beled data for pretraining can be time-intensive; also, it can be detrimental to the performance of the downstream tasks, if the unlabeled data is not aligned with the data distribution for the target tasks. Previous works employed exter- nal supervision in the form of ontologies for selecting appropriate data samples for pretrain- ing, but external supervision can be quite hard to obtain in low-resource domains. In this pa- per, we introduce effective ways to select data from unlabeled corpora of target domains for language model pretraining to improve the per- formances in target entity extraction tasks. Our data selection strategies do not require any ex- ternal supervision. We conduct extensive ex- periments for the task of named entity recog- nition (NER) on seven different domains and show that language models pretrained on tar- get domain unlabeled data obtained using our data selection strategies achieve better perfor- mances compared to those using data selection strategies in previous works that use external supervision. We also show that these pretrained language models using our data selection strate- gies outperform those pretrained on all of the available unlabeled target domain data. 1 Introduction Named entity recognition (NER) (Lample et al., 2016; Nadeau and Sekine, 2007; Finkel and Man- ning, 2009) is the task of extracting entities from a given piece of text. NER is useful for several +Work done while authors were at Adobe Research. natural language processing (NLP) applications such as information extraction (Chiticariu et al., 2013), retrieval (Banerjee et al., 2019), and lan- guage understanding. Over the years, various approaches including rule-based techniques (Far- makiotou et al., 2000), unsupervised learning ap- proaches (Luo et al., 2019), feature-based extrac- tions (Li et al., 2020) have been explored for NER. The recent deep learning-based methods are shown to result in state-of-the-art performances for vari- ous domains (Chiu and Nichols, 2016; Peters et al., 2018; Yadav and Bethard, 2018). Specifically, large pretrained language models such as BERT (Devlin et al., 2019) are widely used for NER, as they mark state-of-the-art perfor- mances (Jia et al., 2019). However, these models re- quire large amounts of annotated data which stands as a bottleneck to the training process, particularly when domains having few labeled data are involved. …

Content truncated to 500 words