Multi-Source Cross-Lingual Model Transfer: Learning What to Share

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3098–3112 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3098 Multi-Source Cross-Lingual Model Transfer: Learning What to Share Xilun Chen†∗ Ahmed Hassan Awadallah‡ Hany Hassan‡ Wei Wang‡ Claire Cardie† †Cornell University Ithaca, NY {xlchen,cardie}@cs.cornell.edu ‡Microsoft Research Redmond, WA {hassanam,hanyh,Wei.Wang}@microsoft.com Abstract Modern NLP applications have enjoyed a great boost utilizing neural networks models. Such deep neural models, however, are not applica- ble to most human languages due to the lack of annotated training data for various NLP tasks. Cross-lingual transfer learning (CLTL) is a viable method for building NLP models for a low-resource target language by lever- aging labeled data from other (source) lan- guages. In this work, we focus on the multi- lingual transfer setting where training data in multiple source languages is leveraged to fur- ther boost target language performance. Unlike most existing methods that rely only on language-invariant features for CLTL, our approach coherently utilizes both language- invariant and language-specific features at in- stance level. Our model leverages adversarial networks to learn language-invariant features, and mixture-of-experts models to dynamically exploit the similarity between the target lan- guage and each individual source language1. This enables our model to learn effectively what to share between various languages in the multilingual setup. Moreover, when coupled with unsupervised multilingual embeddings, our model can operate in a zero-resource set- ting where neither target language training data nor cross-lingual resources are available. Our model achieves significant performance gains over prior art, as shown in an extensive set of experiments over multiple text classifi- cation and sequence tagging tasks including a large-scale industry dataset. 1 Introduction Recent advances in deep learning enabled a wide variety of NLP models to achieve impressive per- formance, thanks in part to the availability of ∗Most work was done while the first author was an intern at Microsoft Research. 1The code is available at https://github.com/ microsoft/Multilingual-Model-Transfer. large-scale annotated datasets. However, such an advantage is not available to most of the world lan- guages since many of them lack the the labeled data necessary for training deep neural nets for a variety of NLP tasks. As it is prohibitive to obtain training data for all languages of interest, cross- lingual transfer learning (CLTL) offers the pos- sibility of learning models for a target language using annotated data from other languages (source languages) (Yarowsky et al., 2001). In this paper, we concentrate on the more challenging unsuper- vised CLTL setting, where no target language la- beled data is used for training.2 Traditionally, most research on CLTL has been devoted to the standard bilingual transfer (BLTL) case where training data comes from a single source language. In practice, however, it is of- ten the case that we have labeled data in a few languages, and would like to be able to utilize all of the data when transferring to other languages. Previous work (McDonald et al., 2011) indeed showed that transferring from multiple source lan- …

Content truncated to 500 words