Trained on 100 million words and still in shape: BERT meets British National Corpus

Findings of the Association for Computational Linguistics: EACL 2023, pages 1954–1974 May 2-6, 2023 ©2023 Association for Computational Linguistics Trained on 100 million words and still in shape: BERT meets British National Corpus David Samuel, Andrey Kutuzov, Lilja Øvrelid and Erik Velldal University of Oslo, Language Technology Group {davisamu, andreku, liljao, erikve}@ifi.uio.no Abstract While modern masked language models (LMs) are trained on ever larger corpora, we here explore the effects of down-scaling training to a modestly-sized but representative, well- balanced, and publicly available English text source – the British National Corpus. We show that pre-training on this carefully curated cor- pus can reach better performance than the orig- inal BERT model. We argue that this type of corpora has great potential as a language mod- eling benchmark. To showcase this potential, we present fair, reproducible and data-efficient comparative studies of LMs, in which we evalu- ate several training objectives and model archi- tectures and replicate previous empirical results in a systematic way. We propose an optimized LM architecture called LTG-BERT. 1 Introduction In the pursuit of state-of-the-art performance, NLP practitioners utilize increasingly larger amounts of data to pre-train language models, making it diffi- cult to disentangle the improvements made by the proposed modeling choices themselves. Instead, our aim is to shift the focus towards more efficient language modeling on a small and standardizable pre-training corpus. We study the data efficiency of current language models on an openly available corpus of approximately 100M words – inciden- tally the estimated amount of words processed by humans before adulthood (Linzen, 2020). The goal of this paper is not to rival the paradigm of ‘massively pre-trained language models’; in- stead we would in this work like to pursue a com- plementary direction of language modeling, which will hopefully lead to more interest in data-efficient language models. In particular, our contribution in this paper is twofold – we show that: 1. 100M words is enough to train a competitive language model that outperforms the downstream performance of the original BERT model. We show that the combination of a well-curated representa- tive corpus, improved LTG-BERT architecture and a better training objective results in a model with stronger linguistic knowledge than the original En- glish BERT pre-trained on 30× larger corpus. Large language models are notoriously data hun- gry, requiring hundreds of gigabytes of raw tex- tual data. This becomes a major obstacle for low- resource languages while also putting a limit to the efficiency of any ‘efficient’ language model. On top of that, the size of web-crawled corpora makes it almost impossible to control their content and to prevent learning from harmful or copyrighted text (Bender et al., 2021). The British National Corpus (BNC; Consortium, 2007) is a 100-million-word reference corpus, manually curated to cover most aspects of 20th century British English. 2. Reproducibility and fair comparison of lan- guage models can be easily achieved by pre- training on the British National Corpus. Massive language models are often pre-trained on nonpublic filtered collections of web-crawled text, …

Content truncated to 500 words