Cross-Lingual Dependency Parsing with Unlabeled Auxiliary Languages

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 372–382 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 372 Cross-lingual Dependency Parsing with Unlabeled Auxiliary Languages Wasi Uddin Ahmad1, Zhisong Zhang2, Xuezhe Ma2 wasiahmad@cs.ucla.edu,{zhisongz,xuezhem}@cs.cmu.edu Kai-Wei Chang1, Nanyun Peng3 kwchang@cs.ucla.edu,npeng@isi.edu 1University of California, Los Angeles, 2Carnegie Mellon University 3University of Southern California Abstract Cross-lingual transfer learning has become an important weapon to battle the unavailabil- ity of annotated resources for low-resource languages. One of the fundamental tech- niques to transfer across languages is learn- ing language-agnostic representations, in the form of word embeddings or contextual en- codings. In this work, we propose to lever- age unannotated sentences from auxiliary lan- guages to help learning language-agnostic rep- resentations. Specifically, we explore adver- sarial training for learning contextual encoders that produce invariant representations across languages to facilitate cross-lingual transfer. We conduct experiments on cross-lingual de- pendency parsing where we train a depen- dency parser on a source language and trans- fer it to a wide range of target languages. Ex- periments on 28 target languages demonstrate that adversarial training significantly improves the overall transfer performances under several different settings. We conduct a careful analy- sis to evaluate the language-agnostic represen- tations resulted from adversarial training. 1 Introduction Cross-lingual transfer, where a model learned from one language is transferred to another, has become an important technique to improve the quality and coverage of natural language process- ing (NLP) tools for languages in the world. This technique has been widely applied in many ap- plications, including part-of-speech (POS) tagging (Kim et al., 2017), dependency parsing (Ma and Xia, 2014), named entity recognition (Xie et al., 2018), entity linking (Sil et al., 2018), coreference resolution (Kundu et al., 2018), and question an- swering (Joty et al., 2017). Noteworthy improve- ments are achieved on low resource language ap- plications due to cross-lingual transfer learning. In this paper, we study cross-lingual transfer for dependency parsing. A dependency parser con- sists of (1) an encoder that transforms an input text sequence into latent representations and (2) a de- coding algorithm that generates the corresponding parse tree. In cross-lingual transfer, most recent approaches assume that the inputs from different languages are aligned into the same embedding space via multilingual word embeddings or multi- lingual contextualized word vectors, such that the parser trained on a source language can be trans- ferred to target languages. However, when train- ing a parser on the source language, the encoder not only learns to embed a sentence but it also car- ries language-specific properties, such as word or- der typology. Therefore, the parser suffers when it is transferred to a language with different language properties. Motivated by this, we study how to train an encoder for generating language-agnostic representations that can be transferred across a wide variety of languages. We propose to utilize unlabeled sentences of one or more auxiliary languages to train an en- coder that learns language-agnostic contextual representations of sentences to facilitate cross- lingual transfer. To utilize …

Content truncated to 500 words