BARThez: a Skilled Pretrained French Sequence-to-Sequence Model

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9369–9390 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 9369 BARThez: a Skilled Pretrained French Sequence-to-Sequence Model Moussa Kamal Eddine École Polytechnique Antoine J.-P. Tixier École Polytechnique Michalis Vazirgiannis École Polytechnique Abstract Inductive transfer learning has taken the en- tire NLP field by storm, with models such as BERT and BART setting new state of the art on countless NLU tasks. However, most of the available models and research have been conducted for English. In this work, we introduce BARThez, the first large-scale pretrained seq2seq model for French. Be- ing based on BART, BARThez is particularly well-suited for generative tasks. We evalu- ate BARThez on five discriminative tasks from the FLUE benchmark and two generative tasks from a novel summarization dataset, Orange- Sum, that we created for this research. We show BARThez to be very competitive with state-of-the-art BERT-based French language models such as CamemBERT and FlauBERT. We also continue the pretraining of a mul- tilingual BART on BARThez’ corpus, and show our resulting model, mBARThez, to sig- nificantly boost BARThez’ generative perfor- mance. Code, data and models are publicly available. 1 Introduction Inductive transfer learning, that is, solving tasks with models that have been pretrained on very large amounts of data, was a game changer in computer vision (Krizhevsky et al., 2012). In NLP, while annotated data are scarce, raw text is virtually un- limited and readily available. It thus emerged that the ability to learn good representations from plain text could greatly improve general natural language understanding. Trained on gigantic amounts of raw data and with hundreds of GPUs, models based on the Trans- former architecture (Vaswani et al., 2017), such as GPT (rad) and BERT (Devlin et al., 2019), have set new state-of-the-art performance in every NLU task. Moreover, users around the world can eas- ily benefit from these improvements, by finetuning the publicly available pretrained models to their specific applications. This also saves considerable amounts of time, resources and energy, compared with training models from scratch. BART (Lewis et al., 2020) combined a BERT- liked bidirectional encoder with a GPT-like forward decoder, and pretrained this seq2seq architecture as a denoising autoencoder with a more general formulation of the masked language modeling ob- jectives of BERT. Since not only BART’s encoder but also its decoder is pretrained, BART excels on tasks involving text generation. While the aforementioned efforts have made great strides, most of the research and resources were dedicated to the English language, despite a few notable exceptions. In this paper, we partly address this limitation by contributing BARThez1, the first pretrained seq2seq model for French. BARThez, based on BART, was pretrained on a very large monolingual French corpus from past re- search that we adapted to suit BART’s specific per- turbation schemes. Unlike already existing BERT- based French language models such as Camem- BERT (Martin et al., 2020) and FlauBERT (Le et al., 2020), BARThez is particularly well-suited for generative tasks. We …

Content truncated to 500 words