On the Usability of Transformers-based Models for a French Question-Answering Task

Proceedings of Recent Advances in Natural Language Processing, pages 244–255 Sep 1–3, 2021. https://doi.org/10.26615/978-954-452-072-4_029 244 On the Usability of Transformers-based models for a French Question-Answering task Oralie Cattan1,2 Christophe Servan1 Sophie Rosset2 1QWANT 61 rue de Villiers, 92200 Neuilly-sur-Seine, France inital.lastname@qwant.com 2Universit´e Paris-Saclay, CNRS, LISN, 91405, Orsay, France lastname@lisn.fr Abstract For many tasks, state-of-the-art results have been achieved with Transformer-based archi- tectures, resulting in a paradigmatic shift in practices from the use of task-specific architec- tures to the fine-tuning of pre-trained language models. The ongoing trend consists in train- ing models with an ever-increasing amount of data and parameters, which requires consider- able resources. It leads to a strong search to im- prove resource efficiency based on algorithmic and hardware improvements evaluated only for English. This raises questions about their usability when applied to small-scale learn- ing problems, for which a limited amount of training data is available, especially for under- resourced languages tasks. The lack of appro- priately sized corpora is a hindrance to apply- ing data-driven and transfer learning-based ap- proaches with strong instability cases. In this paper, we establish a state-of-the-art of the ef- forts dedicated to the usability of Transformer- based models and propose to evaluate these improvements on the question-answering per- formances of French language which have few resources. We address the instability relat- ing to data scarcity by investigating various training strategies with data augmentation, hy- perparameters optimization and cross-lingual transfer. We also introduce a new compact model for French FrALBERT which proves to be competitive in low-resource settings. 1 Introduction Recent advances in the field of Natural Language Processing (NLP) have been made with the devel- opment of transfer learning and the availability of pre-trained language models based on Transformer architectures (Vaswani et al., 2017), such as BERT (Devlin et al., 2019). As they provide contextual- ized semantic representation they contribute both to advance the state-of-the-art on several NLP tasks and also to evolve training practices through the use of fine-tuning. The trend of recent years consists in training large pre-trained language models on ever larger corpora, with an ever-increasing amount of param- eters, which requires considerable computational resources that only a few companies and institu- tions can afford. For example, the base model of BERT with 110 million parameters was pre-trained on 16 gigabytes (GB) of text, while the GPT-3 model (Brown et al., 2020) was pre-trained on 45 terabytes (TB) of text and has 175 billion parame- ters. In fact, deploying ever-larger models raises ques- tions and concerns about the increasing magni- tude of the temporal, financial, and environmen- tal cost of training and usability (Strubell et al., 2019; Moosavi et al., 2020). Typically, due to their resource requirements, these models are trained and deployed for industrial operations on remote servers. This leads to a high use of over-the-air communications, which are particularly resource- intensive (G¨und¨uz et al., 2019). In particular, some NLP applications (speech recognition, speech to text, etc.) have some known problems related to network latency, transmission path difficulties, …

Content truncated to 500 words