A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1703–1714 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1703 A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages Pedro Javier Ortiz Suárez1,2 Laurent Romary1 Benoît Sagot1 1Inria, Paris, France 2Sorbonne Université, Paris, France {pedro.ortiz, benoit.sagot, laurent.romary}@inria.fr Abstract We use the multilingual OSCAR corpus, ex- tracted from Common Crawl via language classification, filtering and cleaning, to train monolingual contextualized word embeddings (ELMo) for five mid-resource languages. We then compare the performance of OSCAR- based and Wikipedia-based ELMo embed- dings for these languages on the part-of- speech tagging and parsing tasks. We show that, despite the noise in the Common-Crawl- based OSCAR data, embeddings trained on OSCAR perform much better than monolin- gual embeddings trained on Wikipedia. They actually equal or improve the current state of the art in tagging and parsing for all five languages. In particular, they also improve over multilingual Wikipedia-based contextual embeddings (multilingual BERT), which al- most always constitutes the previous state of the art, thereby showing that the benefit of a larger, more diverse corpus surpasses the cross- lingual benefit of multilingual embedding ar- chitectures. 1 Introduction One of the key elements that has pushed the state of the art considerably in neural NLP in recent years has been the introduction and spread of transfer learning methods to the field. These methods can normally be classified in two categories according to how they are used: • Feature-based methods, which involve pre- training real-valued vectors (“embeddings”) at the word, sentence, or paragraph level; and us- ing them in conjunction with a specific archi- tecture for each individual downstream task. • Fine-tuning methods, which introduce a mini- mal number of task-specific parameters, and instead copy the weights from a pre-trained network and then tune them to a particular downstream task. Embeddings or language models can be divided into fixed, meaning that they generate a single rep- resentation for each word in the vocabulary; and contextualized, meaning that a representation is generated based on both the word and its surround- ing context, so that a single word can have multiple representations, each one depending on how it is used. In practice, most fixed embeddings are used as feature-based models. The most notable examples are word2vec (Mikolov et al., 2013), GloVe (Pen- nington et al., 2014) and fastText (Mikolov et al., 2018). All of them are extensively used in a vari- ety of applications nowadays. On the other hand, contextualized word representations and language models have been developed using both feature- based architectures, the most notable examples be- ing ELMo and Flair (Peters et al., 2018; Akbik et al., 2018), and transformer based architectures, that are commonly used in a fine-tune setting, as is the case of GPT-1, GPT-2 (Radford et al., 2018, 2019), BERT and its derivatives (Devlin et al., 2018; Liu et al., 2019; Lan et al., 2019) and more recently T5 (Raffel et al., 2019). All of them have repeatedly improved …

Content truncated to 500 words