Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2832–2838 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Cross-Lingual Transfer Learning for POS Tagging without Cross-Lingual Resources Joo-Kyung Kim†, Young-Bum Kim‡, Ruhi Sarikaya‡, Eric Fosler-Lussier† †The Ohio State University, Columbus, OH 43210, USA ‡Amazon Alexa, Seattle, WA 98121, USA Abstract Training a POS tagging model with cross- lingual transfer learning usually requires linguistic knowledge and resources about the relation between the source language and the target language. In this pa- per, we introduce a cross-lingual transfer learning model for POS tagging without ancillary resources such as parallel cor- pora. The proposed cross-lingual model utilizes a common BLSTM that enables knowledge transfer from other languages, and private BLSTMs for language-specific representations. The cross-lingual model is trained with language-adversarial train- ing and bidirectional language modeling as auxiliary objectives to better represent language-general information while not losing the information about a specific tar- get language. Evaluating on POS datasets from 14 languages in the Universal De- pendencies corpus, we show that the pro- posed transfer learning model improves the POS tagging performance of the tar- get languages without exploiting any lin- guistic knowledge between the source lan- guage and the target language. 1 Introduction Bidirectional Long Short-Term Memory (BLSTM) based models (Graves and Schmid- huber, 2005), along with word embeddings and character embeddings, have shown competitive performance on Part-of-Speech (POS) tagging given sufficient amount of training examples (Ling et al., 2015; Lample et al., 2016; Plank et al., 2016; Yang et al., 2017). Given insufficient training examples, we can improve the POS tagging performance by cross- lingual POS tagging, which exploits affluent POS tagging corpora from other source languages. This approach usually requires linguistic knowledge or resources about the relation between the source language and the target language such as paral- lel corpora (T¨ackstr¨om et al., 2013; Duong et al., 2013; Kim et al., 2015a; Zhang et al., 2016), mor- phological analyses (Hana et al., 2004), dictionar- ies (Wisniewski et al., 2014), and gaze features (Barrett et al., 2016). Given no linguistic resources between the source language and the target language, transfer learning methods can be utilized instead. Trans- fer learning for cross-lingual cases is a type of transductive transfer learning, where the input do- mains of the source and the target are different (Pan and Yang, 2010) since each language has its own vocabulary space. When the input space is the same, lower layers of hierarchical models can be shared for knowledge transfer (Collobert et al., 2011; Kim et al., 2015b; Yang et al., 2017), but that approach is not directly applicable when the input spaces differ. Yang et al. (2017) used shared character em- beddings for different languages as a cross-lingual transfer method while using different word em- beddings for different languages. Although the ap- proach showed improved performance on Named Entity Recognition, it is limited to character-level representation transfer and it is not applicable for knowledge transfer between languages without overlapped alphabets. In this …

Content truncated to 500 words