On the importance of pre-training data volume for compact language models

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7853–7858, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7853 On the importance of pre-training data volume for compact language models Vincent Micheli Illuin Technology / EPFL vincent.micheli@illuin.tech Martin D’Hoffschmidt Illuin Technology martin.dhoffschmidt@illuin.tech Franc¸ois Fleuret UNIGE francois.fleuret@unige.ch Abstract Recent advances in language modeling have led to computationally intensive and resource- demanding state-of-the-art models. In an ef- fort towards sustainable practices, we study the impact of pre-training data volume on com- pact language models. Multiple BERT-based models are trained on gradually increasing amounts of French text. Through fine-tuning on the French Question Answering Dataset (FQuAD), we observe that well-performing models are obtained with as little as 100 MB of text. In addition, we show that past critically low amounts of pre-training data, an interme- diate pre-training step on the task-specific cor- pus does not yield substantial improvements. 1 Introduction Over the past year, pre-trained language mod- els have become the norm in Natural Language Processing. These large-scale Transformer-based (Vaswani et al., 2017) networks considerably ad- vanced the state-of-the-art in language understand- ing (Devlin et al., 2019) via a two-step process: self-supervised learning on a vast text corpus fol- lowed by fine-tuning on a specific downstream task. Following these advances, the ongoing trend has been to build bigger models with an ever-increasing amount of data (Liu et al., 2019; Raffel et al., 2020; Radford et al., 2019; Brown et al., 2020). How- ever, pre-training models with billions of param- eters over hundreds of gigabytes of text requires tremendous computational resources that only a few companies and institutions can afford. Besides, many languages and specific corpora (e.g. legal, scientific) are currently under-resourced. Hence, our goal is to explore model architectures and data volumes lowering the entry barrier to new research and practical applications. We conduct experiments on French corpora in order to release the first French compact language models and to illustrate the training process in an- other language than English. Furthermore, we con- sider the question answering task since compact models may find their purpose in low-latency/fault- tolerant information retrieval systems. 2 Problem statement We intend to study the impact of pre-training data volume when training compact bidirectional Trans- formers (Devlin et al., 2019). We assume a scarce resources setting, both in terms of data and com- puting power. Two key aspects are explored: • The amount of pre-training data required to train high-performing compact language mod- els. • The importance of corpus-specific MLM be- fore fine-tuning. We use the French part of the OSCAR corpora (Ortiz Suarez et al., 2019) for pre-training and the FQuAD dataset 1 (d’Hoffschmidt et al., 2020) for machine reading comprehension fine-tuning. More- over, the models under consideration are based on the CamemBERT (Martin et al., 2020) language model. 3 Related work A wealth of work has recently been released (Ganesh et al., 2020) on compressing Transformer- based models (Vaswani et al., 2017; Devlin et al., 2019) through the pre-training of compact models (Turc …

Content truncated to 500 words