Choosing Transfer Languages for Cross-Lingual Learning

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3125–3135 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3125 Choosing Transfer Languages for Cross-Lingual Learning Yu-Hsiang Lin∗, Chian-Yu Chen∗, Jean Lee∗, Zirui Li∗, Yuyan Zhang∗, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell†, Graham Neubig Language Technologies Institute, Carnegie Mellon University †National Research Council, Canada Abstract Cross-lingual transfer, where a high-resource transfer language is used to improve the accu- racy of a low-resource task language, is now an invaluable tool for improving performance of natural language processing (NLP) on low- resource languages. However, given a particu- lar task language, it is not clear which language to transfer from, and the standard strategy is to select languages based on ad hoc criteria, usu- ally the intuition of the experimenter. Since a large number of features contribute to the suc- cess of cross-lingual transfer (including phylo- genetic similarity, typological properties, lex- ical overlap, or size of available data), even the most enlightened experimenter rarely con- siders all these factors for the particular task at hand. In this paper, we consider this task of automatically selecting optimal transfer lan- guages as a ranking problem, and build mod- els that consider the aforementioned features to perform this prediction. In experiments on representative NLP tasks, we demonstrate that our model predicts good transfer languages much better than ad hoc baselines consider- ing single features in isolation, and glean in- sights on what features are most informative for each different NLP tasks, which may in- form future ad hoc selection even without use of our method.1 1 Introduction A common challenge in applying natural language processing (NLP) techniques to low-resource lan- guages is the lack of training data in the languages in question. It has been demonstrated that through cross-lingual transfer, it is possible to leverage one or more similar high-resource languages to im- prove the performance on the low-resource lan- guages in several NLP tasks, including machine ∗Equal contribution 1Code, data, and pre-trained models are available at https://github.com/neulab/langrank score(Ltf,1, Ltk) score(Ltf,2, Ltk) ... Ltf,1: Transfer Language 1 Ltk: Task Language Ltk: Task Language Transfer Learning Transfer Learning Generate Training Data Train Transfer Language Ranker Learning to Rank ... ... ... Ltf,2: Transfer Language 2 NLP Model 1 score(Ltf,1, Ltk) NLP Model 2 score(Ltf,2, Ltk) Transfer Language Ranker ... Figure 1: Workflow of learning to select the transfer languages for an NLP task: (1) train a set of NLP models with all available transfer languages and collect evaluation scores, (2) train a ranking model to predict the top transfer languages. translation (Zoph et al., 2016; Johnson et al., 2017; Nguyen and Chiang, 2017; Neubig and Hu, 2018), parsing (T¨ackstr¨om et al., 2012; Ammar et al., 2016; Ahmad et al., 2018; Ponti et al., 2018), part- of-speech or morphological tagging (T¨ackstr¨om et al., 2013; Cotterell and Heigold, 2017; Malaviya et al., 2018; Plank and Agi´c, 2018), named entity recognition (Zhang et al., …

Content truncated to 500 words