Using Similarity Measures to Select Pretraining Data for NER

Proceedings of NAACL-HLT 2019, pages 1460–1470 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1460 Using Similarity Measures to Select Pretraining Data for NER Xiang Dai1,2 Sarvnaz Karimi1 Ben Hachey2,3 Cecile Paris1 1CSIRO Data61, Sydney, Australia 2University of Sydney, Sydney, Australia 3Digital Health CRC, Sydney, Australia {dai.dai,sarvnaz.karimi,cecile.paris}@csiro.au ben.hachey@gmail.com Abstract Word vectors and Language Models (LMs) pretrained on a large amount of unlabelled data can dramatically improve various Natural Lan- guage Processing (NLP) tasks. However, the measure and impact of similarity between pre- training data and target task data are left to in- tuition. We propose three cost-effective mea- sures to quantify different aspects of similar- ity between source pretraining and target task data. We demonstrate that these measures are good predictors of the usefulness of pretrained models for Named Entity Recognition (NER) over 30 data pairs. Results also suggest that pretrained LMs are more effective and more predictable than pretrained word vectors, but pretrained word vectors are better when pre- training data is dissimilar. 1 Introduction Modern neural architectures for NLP are highly effective when provided a large amount of labelled training data (Zhang et al., 2015; Conneau et al., 2017; Bowman et al., 2015). However, a large la- belled data set is not always readily accessible due to the high cost of expertise needed for labelling or even due to legal barriers. Researchers working on such tasks usually spend a considerable amount of effort and resources on collecting useful exter- nal data sources and investigating how to transfer knowledge to their target tasks (Qi et al., 2009; Kim et al., 2017). Recent transfer learning tech- niques make the most of limited labelled data by incorporating word vectors or LMs pretrained on a large amount of unlabelled data. This produces dramatic improvements over a range of NLP tasks where appropriate unlabelled data is available (Pe- ters et al., 2017, 2018; Akbik et al., 2018; Devlin et al., 2019). However, there is still a lack of systematic study on how to select appropriate data to pretrain word vectors or LMs. We observe a range of heuristic strategies in the literature: (1) collecting a large amount of generic data, e.g., web crawl (Penning- ton et al., 2014; Mikolov et al., 2018); (2) selecting data from a similar field (the subject matter of the content being discussed), e.g., biology (Chiu et al., 2016; Karimi et al., 2017); and, (3) selecting data from a similar tenor (the participants in the dis- course, their relationships to each other, and their purposes), e.g., Twitter, or online forums (Li et al., 2017; Chronopoulou et al., 2019). In all these set- tings, the decision is based on heuristics and varies according to the individual’s experience. We also conducted a pilot study that suggests that the prac- titioner’s intuition is to prioritise field over tenor (see Section 3). Our overarching goal is to develop a cost- effective approach that, given a NER data set, nominates the most suitable source data to pretrain word vectors …

Content truncated to 500 words