On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing

Proceedings of NAACL-HLT 2019, pages 2440–2452 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2440 On Difficulties of Cross-Lingual Transfer with Order Differences: A Case Study on Dependency Parsing Wasi Uddin Ahmad1∗, Zhisong Zhang2∗, Xuezhe Ma2 wasiahmad@cs.ucla.edu,{zhisongz,xuezhem}@cs.cmu.edu Eduard Hovy2, Kai-Wei Chang1, Nanyun Peng3† ehovy@cs.cmu.edu,kwchang@cs.ucla.edu,npeng@isi.edu 1University of California, Los Angeles, 2Carnegie Mellon University 3University of Southern California Abstract Different languages might have different word orders. In this paper, we investigate cross- lingual transfer and posit that an order- agnostic model will perform better when trans- ferring to distant foreign languages. To test our hypothesis, we train dependency parsers on an English corpus and evaluate their transfer per- formance on 30 other languages. Specifically, we compare encoders and decoders based on Recurrent Neural Networks (RNNs) and mod- ified self-attentive architectures. The former relies on sequential information while the lat- ter is more flexible at modeling word order. Rigorous experiments and detailed analysis shows that RNN-based architectures transfer well to languages that are close to English, while self-attentive models have better overall cross-lingual transferability and perform espe- cially well on distant languages. 1 Introduction Cross-lingual transfer, which transfers models across languages, has tremendous practical value. It reduces the requirement of annotated data for a target language and is especially useful when the target language is lack of resources. Recently, this technique has been applied to many NLP tasks such as text categorization (Zhou et al., 2016a), tagging (Kim et al., 2017), dependency parsing (Guo et al., 2015, 2016) and machine translation (Zoph et al., 2016). Despite the preliminary suc- cess, transferring across languages is challenging as it requires understanding and handling differ- ences between languages at levels of morphology, syntax, and semantics. It is especially difficult to learn invariant features that can robustly transfer to distant languages. ∗Equal contribution. Listed by alphabetical order. †Corresponding author. Prior work on cross-lingual transfer mainly fo- cused on sharing word-level information by lever- aging multi-lingual word embeddings (Xiao and Guo, 2014; Guo et al., 2016; Sil et al., 2018). However, words are not independent in sentences; their combinations form larger linguistic units, known as context. Encoding context information is vital for many NLP tasks, and a variety of ap- proaches (e.g., convolutional neural networks and recurrent neural networks) have been proposed to encode context as a high-level feature for down- stream tasks. In this paper, we study how to transfer generic contextual information across lan- guages. For cross-language transfer, one of the key chal- lenges is the variation in word order among differ- ent languages. For example, the Verb-Object pat- tern in English can hardly be found in Japanese. This challenge should be taken into considera- tion in model design. RNN is a prevalent family of models for many NLP tasks and has demon- strated compelling performances (Mikolov et al., 2010; Sutskever et al., 2014; Peters et al., 2018). However, its sequential nature makes it heavily re- liant on word order information, which exposes to the risk of encoding language-specific order in- …

Content truncated to 500 words