CamemBERT: a Tasty French Language Model

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7203–7219 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7203 CamemBERT: a Tasty French Language Model Louis Martin∗1,2,3 Benjamin Muller∗2,3 Pedro Javier Ortiz Suárez∗2,3 Yoann Dupont3 Laurent Romary2 Éric Villemonte de la Clergerie2 Djamé Seddah2 Benoît Sagot2 1Facebook AI Research, Paris, France 2Inria, Paris, France 3Sorbonne Université, Paris, France louismartin@fb.com {benjamin.muller, pedro.ortiz, laurent.romary, eric.de_la_clergerie, djame.seddah, benoit.sagot}@inria.fr yoa.dupont@gmail.com Abstract Pretrained language models are now ubiqui- tous in Natural Language Processing. Despite their success, most available models have ei- ther been trained on English data or on the con- catenation of data in multiple languages. This makes practical use of such models—in all lan- guages except English—very limited. In this paper, we investigate the feasibility of train- ing monolingual Transformer-based language models for other languages, taking French as an example and evaluating our language models on part-of-speech tagging, dependency parsing, named entity recognition and natural language inference tasks. We show that the use of web crawled data is preferable to the use of Wikipedia data. More surprisingly, we show that a relatively small web crawled dataset (4GB) leads to results that are as good as those obtained using larger datasets (130+GB). Our best performing model CamemBERT reaches or improves the state of the art in all four down- stream tasks. 1 Introduction Pretrained word representations have a long history in Natural Language Processing (NLP), from non- contextual (Brown et al., 1992; Ando and Zhang, 2005; Mikolov et al., 2013; Pennington et al., 2014) to contextual word embeddings (Peters et al., 2018; Akbik et al., 2018). Word representations are usu- ally obtained by training language model architec- tures on large amounts of textual data and then fed as an input to more complex task-specific architec- tures. More recently, these specialized architectures have been replaced altogether by large-scale pre- trained language models which are fine-tuned for each application considered. This shift has resulted in large improvements in performance over a wide ∗Equal contribution. Order determined alphabetically. range of tasks (Devlin et al., 2019; Radford et al., 2019; Liu et al., 2019; Raffel et al., 2019). These transfer learning methods exhibit clear advantages over more traditional task-specific ap- proaches. In particular, they can be trained in an unsupervized manner, thereby taking advantage of the information contained in large amounts of raw text. Yet they come with implementation chal- lenges, namely the amount of data and computa- tional resources needed for pretraining, which can reach hundreds of gigabytes of text and require hundreds of GPUs (Yang et al., 2019; Liu et al., 2019). This has limited the availability of these state-of-the-art models to the English language, at least in the monolingual setting. This is particularly inconvenient as it hinders their practical use in NLP systems. It also prevents us from investigating their language modelling capacity, for instance in the case of morphologically rich languages. Although multilingual models give remarkable results, they are often larger, and their results, as we will observe …

Content truncated to 500 words