Low-Resource Parsing with Crosslingual Contextualized Representations

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 304–315 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 304 Low-Resource Parsing with Crosslingual Contextualized Representations Phoebe Mulcaire~⇤ Jungo Kasai~⇤ Noah A. Smith~} ~Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, USA }Allen Institute for Artificial Intelligence, Seattle, WA, USA {pmulc,jkasai,nasmith}@cs.washington.edu Abstract Despite advances in dependency parsing, lan- guages with small treebanks still present chal- lenges. We assess recent approaches to multilingual contextual word representations (CWRs), and compare them for crosslingual transfer from a language with a large tree- bank to a language with a small or nonexistent treebank, by sharing parameters between lan- guages in the parser itself. We experiment with a diverse selection of languages in both sim- ulated and truly low-resource scenarios, and show that multilingual CWRs greatly facilitate low-resource dependency parsing even with- out crosslingual supervision such as dictionar- ies or parallel text. Furthermore, we exam- ine the non-contextual part of the learned lan- guage models (which we call a “decontextual probe”) to demonstrate that polyglot language models better encode crosslingual lexical cor- respondence compared to aligned monolingual language models. This analysis provides fur- ther evidence that polyglot training is an effec- tive approach to crosslingual transfer. 1 Introduction Dependency parsing has achieved new states of the art using distributed word representations in neural networks, trained with large amounts of an- notated data (Dozat and Manning, 2017; Dozat et al., 2017; Ma et al., 2018; Che et al., 2018). However, many languages are low-resource, with small or no treebanks, which presents a severe challenge in developing accurate parsing systems in those languages. One way to address this prob- lem is with a crosslingual solution that makes use of a language with a large treebank and raw text in both languages. The hypothesis behind this approach is that, although each language is unique, different languages manifest similar char- ⇤ Equal contribution. Random order. acteristics (e.g., morphological, lexical, syntac- tic) which can be exploited by training a single polyglot model with data from multiple languages (Ammar, 2016). Recent work has extended contextual word rep- resentations (CWRs) multilingually either by train- ing a polyglot language model (LM) on a mix- ture of data from multiple languages (joint train- ing approach; Mulcaire et al., 2019; Lample and Conneau, 2019) or by aligning multiple monolin- gual language models crosslingually (retrofitting approach; Schuster et al., 2019; Aldarmaki and Diab, 2019). These multilingual representations have been shown to facilitate crosslingual trans- fer on several tasks, including Universal Depen- dencies parsing and natural language inference. In this work, we assess these two types of meth- ods by using them for low-resource dependency parsing, and discover that the joint training ap- proach substantially outperforms the retrofitting approach. We further apply multilingual CWRs produced by the joint training approach to diverse languages, and show that it is still effective in transfer between distant languages, though we find that phylogenetically related source languages are generally more helpful. …

Content truncated to 500 words