Self Promotion in US Congressional Tweets

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 448–462 June 6–11, 2021. ©2021 Association for Computational Linguistics 448 When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models Benjamin Muller†* Antonios Anastasopoulos‡ Benoît Sagot† Djamé Seddah† †Inria, Paris, France *Sorbonne Université, Paris, France ‡Department of Computer Science, George Mason University, USA firstname.lastname@inria.fr antonis@gmu.edu Abstract Transfer learning based on pretraining lan- guage models on a large amount of raw data has become a new norm to reach state-of-the- art performance in NLP. Still, it remains unclear how this approach should be applied for unseen languages that are not covered by any available large-scale multilingual language model and for which only a small amount of raw data is generally available. In this work, by compar- ing multilingual and monolingual models, we show that such models behave in multiple ways on unseen languages. Some languages greatly benefit from transfer learning and behave simi- larly to closely related high resource languages whereas others apparently do not. Focusing on the latter, we show that this failure to trans- fer is largely related to the impact of the script used to write such languages. We show that transliterating those languages significantly im- proves the potential of large-scale multilingual language models on downstream tasks. This result provides a promising direction towards making these massively multilingual models useful for a new set of unseen languages.1 1 Introduction Language models are now a new standard to build state-of-the-art Natural Language Process- ing (NLP) systems. In the past year, monolingual language models have been released for more than 20 languages including Arabic, French, German, and Italian (Antoun et al., 2020; Martin et al., 2020; de Vries et al., 2019; Cañete et al., 2020; Kuratov and Arkhipov, 2019; Schweter, 2020, inter alia). Additionally, large-scale multilingual models cov- ering more than 100 languages are now available (XLM-R by Conneau et al. (2020) and mBERT by Devlin et al. (2019)). Still, most of the 6500+ spoken languages in the world (Hammarström, 2016) are not covered—remaining unseen—by 1Code available at https://github.com/benjami n-mlr/mbert-unseen-languages.git those models. Even languages with millions of na- tive speakers like Sorani Kurdish (about 7 million speakers in the Middle East) or Bambara (spoken by around 5 million people in Mali and neighbor- ing countries) are not covered by any available language models at the time of writing. Even if training multilingual models that cover more languages and language varieties is tempting, the curse of multilinguality (Conneau et al., 2020) makes it an impractical solution, as it would require to train ever larger models. Furthermore, as shown by Wu and Dredze (2020), large-scale multilingual language models are sub-optimal for languages that are under-sampled during pretraining. In this paper, we analyze task and language adap- tation experiments to get usable language model- based representations for under-studied low re- source languages. We run experiments on 15 ty- pologically diverse languages on three NLP tasks: part-of-speech (POS) …

Content truncated to 500 words