Automatic Document Selection for Efficient Encoder Pretraining

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9522 - 9530 December 7-11, 2022 ©2022 Association for Computational Linguistics Automatic Document Selection for Efficient Encoder Pretraining Yukun Feng1 Patrick Xia1 Benjamin Van Durme1 João Sedoc2 1Johns Hopkins University 2New York University {yfeng55, paxia, vandurme}@jhu.edu, jsedoc@stern.nyu.edu Abstract Building pretrained language models is con- sidered expensive and data-intensive, but must we increase dataset size to achieve better performance? We propose an alternative to larger training sets by automatically identify- ing smaller yet domain-representative subsets. We extend Cynical Data Selection, a statistical sentence scoring method that conditions on a representative target domain corpus. As an ex- ample, we treat the OntoNotes corpus as a tar- get domain and pretrain a RoBERTa-like en- coder from a cynically selected subset of the Pile. On both perplexity and across several downstream tasks in the target domain, it con- sistently outperforms random selection with 20x less data, 3x fewer training iterations, and 2x less estimated cloud compute cost, validat- ing the recipe of automatic document selection for LM pretraining. 1 Introduction Large pretrained language models have achieved state-of-the-art performance in NLP tasks (Devlin et al., 2019; Liu et al., 2019, i.a.). These studies find that increasing pretraining data size usually leads to better task performance. For many tasks, additional task (in-domain) data helps improve the performance further (Gururangan et al., 2020; Dery et al., 2021; Li et al., 2022). Several studies have found that directly pretraining on task data is more effective : science texts (Beltagy et al., 2019), tweets (Nguyen et al., 2020), legal texts (Chalkidis et al., 2020) or code (Tabassum et al., 2020; Chen et al., 2021). Notably, these domains are known a priori, and identifying data sources for curation is straightforward. In other instances where the domain is less clear, like “offensive online content” (Bai et al., 2021), more complicated data sampling is employed to guess at the desired data distribution suitable for training a downstream classifier. To address such scenarios, we propose automat- ically identifying relevant domain-specific train- 87.45 86.25 85.40 Score Pile ~ 1250GB Random ~ 60GB Manual ~ 30GB Cynical ~ 2.5 GB Figure 1: This figure highlights the efficiency of the au- tomatic cynical selection of documents in the target do- main. Scores are averaged from 8 Edge Probing tasks. Cynically selected 2.5GB data achieves the best score. ing data for a large corpus and subsequently pre- training a model on the selected data. Specifi- cally, we use Cynical Data Selection (Axelrod, 2017), an approach that advanced Moore-Lewis sampling (Moore and Lewis, 2010), to select data from the Pile dataset (Gao et al., 2021). This auto- matic selection method can include possibly over- looked yet relevant documents from domains that may not be too close to the target domain. Figure 1 illustrates this method which achieves higher per- formance on tasks in the target domain by using only 2.5GB (0.5%) of cynically selected data. Specifically, we experiment with pretraining en- coders with varying amounts …

Content truncated to 500 words