Polyglot Contextual Representations Improve Crosslingual Transfer

Proceedings of NAACL-HLT 2019, pages 3912–3918 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3912 Polyglot Contextual Representations Improve Crosslingual Transfer Phoebe Mulcaire♥ Jungo Kasai♥ Noah A. Smith♥♦ ♥Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA ♦Allen Institute for Artificial Intelligence, Seattle, WA, USA {pmulc,jkasai,nasmith}@cs.washington.edu Abstract We introduce Rosita, a method to pro- duce multilingual contextual word represen- tations by training a single language model on text from multiple languages. Our method combines the advantages of contex- tual word representations with those of mul- tilingual representation learning. We produce language models from dissimilar language pairs (English/Arabic and English/Chinese) and use them in dependency parsing, seman- tic role labeling, and named entity recognition, with comparisons to monolingual and non- contextual variants. Our results provide fur- ther evidence for the benefits of polyglot learn- ing, in which representations are shared across multiple languages. 1 Introduction State-of-the-art methods for crosslingual transfer make use of multilingual word embeddings, and much research has explored methods that align vector spaces for words in different languages (Faruqui and Dyer, 2014; Upadhyay et al., 2016; Ruder et al., 2017). On the other hand, contextual word representations (CWR) extracted from lan- guage models (LMs) have advanced the state of the art beyond what was achieved with word type representations on many monolingual NLP tasks (Peters et al., 2018). Thus, the question arises: can contextual word representations benefit from multilinguality? We introduce a method to produce multilin- gual CWR by training a single “polyglot” lan- guage model on text in multiple languages. As our work is a multilingual extension of ELMo (Pe- ters et al., 2018), we call it Rosita (after a bilin- gual character from Sesame Street). Our hypothe- sis is that, although each language is unique, dif- ferent languages manifest similar characteristics (e.g., morphological, lexical, syntactic) which can be exploited by training a single model with data from multiple languages (Ammar, 2016). Previ- ous work has shown this to be true to some de- gree in the context of syntactic dependency pars- ing (Ammar et al., 2016), semantic role label- ing (Mulcaire et al., 2018), named entity recog- nition (Xie et al., 2018), and language modeling for phonetic sequences (Tsvetkov et al., 2016) and for speech recognition (Ragni et al., 2016). Re- cently, de Lhoneux et al. (2018) showed that pa- rameter sharing between languages can improve performance in dependency parsing, but the ef- fect is variable, depending on the language pair and the parameter sharing strategy. Other recent work also reported that concatenating data from different languages can hurt performance in de- pendency parsing (Che et al., 2018). These mixed results suggest that while crosslingual transfer in neural network models is a promising direction, the best blend of polyglot and language-specific elements may depend on the task and architecture. However, we find overall contextual representa- tions from polyglot language models succeed in a range of settings, even where multilingual word type embeddings do not, and …

Content truncated to 500 words