Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3211–3223 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3211 Cross-Lingual Syntactic Transfer through Unsupervised Adaptation of Invertible Projections Junxian He1, Zhisong Zhang1, Taylor Berg-Kirkpatrick2, and Graham Neubig1 1Language Technologies Institute, Carnegie Mellon University 2Department of Computer Science and Engineering, University of California San Diego {junxianh,zhisongz,gneubig}@cs.cmu.edu, tberg@eng.ucsd.edu Abstract Cross-lingual transfer is an effective way to build syntactic analysis tools in low-resource languages. However, transfer is difficult when transferring to typologically distant languages, especially when neither annotated target data nor parallel corpora are available. In this pa- per, we focus on methods for cross-lingual transfer to distant languages and propose to learn a generative model with a structured prior that utilizes labeled source data and un- labeled target data jointly. The parameters of source model and target model are softly shared through a regularized log likelihood ob- jective. An invertible projection is employed to learn a new interlingual latent embedding space that compensates for imperfect cross- lingual word embedding input. We evaluate our method on two syntactic tasks: part-of- speech (POS) tagging and dependency pars- ing. On the Universal Dependency Treebanks, we use English as the only source corpus and transfer to a wide range of target languages. On the 10 languages in this dataset that are distant from English, our method yields an average of 5.2% absolute improvement on POS tagging and 8.3% absolute improvement on dependency parsing over a direct trans- fer method using state-of-the-art discrimina- tive models.1 1 Introduction Current top performing systems on syntactic anal- ysis tasks such as part-of-speech (POS) tagging and dependency parsing rely heavily on large- scale annotated data (Huang et al., 2015; Dozat and Manning, 2017; Ma et al., 2018). How- ever, because creating syntactic treebanks is an ex- pensive and time consuming task, annotated data is scarce for many languages. Prior work has 1Code is available at https://github.com/jxhe/ cross-lingual-struct-flow. 0.4 0.5 0.6 0.7 0.8 language distance to English 30 40 50 60 70 80 90 POS Tagging Accuracy% nl sv da no es fr pt it bg he hr hi tr ko id ja zh fa ar nearby languages distant languages 0.4 0.5 0.6 0.7 0.8 language distance to English 30 40 50 60 70 80 90 Dependency Parsing UAS% nl sv da no es fr pt itbg he hr hi tr ko id ja zh fa ar nearby languages distant languages Figure 1: Left: POS tagging transfer accuracy of the Bidirectional LSTM-CRF model, Right: Depen- dency parsing transfer UAS of the “SelfAtt-Graph” model (Ahmad et al., 2019). These models are trained on the labeled English corpus and directly evaluated on different target languages. The x-axis represents language distance to English (details in Section 2.1). Both models take pre-trained cross-lingual word em- beddings as input. The parsing model also uses gold universal POS tags. demonstrated the efficacy of cross-lingual learn- ing methods (Guo et al., 2015; Tiedemann, 2015; Guo et al., 2016; Zhang et …

Content truncated to 500 words