Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification

Proceedings of NAACL-HLT 2016, pages 692–702, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Bilingual Word Embeddings from Parallel and Non-parallel Corpora for Cross-Language Text Classification Aditya Mogadala Institute AIFB Karlsruhe Institute of Technology Karlsruhe, Germany aditya.mogadala@kit.edu Achim Rettinger Institute AIFB Karlsruhe Institute of Technology Karlsruhe, Germany rettinger@kit.edu Abstract In many languages, sparse availability of re- sources causes numerous challenges for tex- tual analysis tasks. Text classification is one of such standard tasks that is hindered due to lim- ited availability of label information in low- resource languages. Transferring knowledge (i.e. label information) from high-resource to low-resource languages might improve text classification as compared to the other ap- proaches like machine translation. We intro- duce BRAVE (Bilingual paRAgraph VEctors), a model to learn bilingual distributed repre- sentations (i.e. embeddings) of words with- out word alignments either from sentence- aligned parallel or label-aligned non-parallel document corpora to support cross-language text classification. Empirical analysis shows that classification models trained with our bilingual embeddings outperforms other state- of-the-art systems on three different cross- language text classification tasks. 1 Introduction The availability of language-specific annotated re- sources is crucial for the efficiency of natural lan- guage processing tasks. Still, many languages lack rich annotated resources that support various tasks such as part-of-speech tagging, dependency parsing and text classification. While the growth of multi- lingual information on the web has provided an op- portunity to build these missing annotated resources, but still lots of manual effort is required to achieve high quality resources for every language separately. Another possibility is to utilize the unlabeled data present in those languages or transfer knowl- edge from annotation-rich languages. For the first alternative, recent advancements made in learning monolingual distributed representations of words (Mikolov et al., 2013a; Pennington et al., 2014; Levy and Goldberg, 2014) (i.e. monolin- gual word embeddings) capturing syntactic and se- mantic information in an unsupervised manner was useful in numerous NLP tasks (Collobert et al., 2011). However, this may not be sufficient for several other tasks such as cross-language informa- tion retrieval (Grefenstette, 2012), cross-language word semantic similarity (Vuli´c and Moens, 2014), cross-language text classification (CLTC, hence- forth) (Klementiev et al., 2012; Xiao and Guo, 2013; Prettenhofer and Stein, 2010; Tang and Wan, 2014) and machine translation (Zhao et al., 2015) due to irregularities across languages. In these kind of sce- narios, transfer of knowledge can be useful. Several approaches (Hermann and Blunsom, 2014; Sarath Chandar et al., 2014; Gouws et al., 2015; Coulmance et al., 2015) tried to induce monolingual distributed representations into a lan- guage independent space (i.e. bilingual or multilin- gual word embeddings) by jointly training on pair of languages. Although the overall goal of these approaches is to capture linguistic regularities in words that share same semantic and syntactic space across languages, they differ in their implementa- tion. One set of methods either perform offline alignment of trained monolingual embeddings or jointly-train both monolingual and cross-lingual ob- jectives, while the other set uses only cross-lingual objective. Jointly-trained …

Content truncated to 500 words