Cross-lingual Distillation for Text Classification

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1415–1425 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1130 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1415–1425 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1130 Cross-lingual Distillation for Text Classification Ruochen Xu Carnegie Mellon Universit ruochenx@cs.cmu.edu Yiming Yang Carnegie Mellon Universit yiming@cs.cmu.edu Abstract Cross-lingual text classification(CLTC) is the task of classifying documents written in different languages into the same tax- onomy of categories. This paper presents a novel approach to CLTC that builds on model distillation, which adapts and ex- tends a framework originally proposed for model compression. Using soft proba- bilistic predictions for the documents in a label-rich language as the (induced) super- visory labels in a parallel corpus of docu- ments, we train classifiers successfully for new languages in which labeled training data are not available. An adversarial fea- ture adaptation technique is also applied during the model training to reduce dis- tribution mismatch. We conducted experi- ments on two benchmark CLTC datasets, treating English as the source language and German, French, Japan and Chinese as the unlabeled target languages. The proposed approach had the advantageous or comparable performance of the other state-of-art methods. 1 Introduction The availability of massive multilingual data on the Internet makes cross-lingual text classification (CLTC) increasingly important. The task is de- fined as to classify documents in different lan- guages using the same taxonomy of predefined categories. CLTC systems build on supervised machine learning require a sufficiently amount of labeled training data for every domain of interest in each language. But in reality, labeled data are not evenly distributed among languages and across domains. English, for example, is a label-rich lan- guage in the domains of news stories, Wikipedia pages and reviews of hotels, products, etc. But many other languages do not necessarily have such rich amounts of labeled data. This leads to an open challenge in CLTC, i.e., how can we effec- tively leverage the trained classifiers in a label-rich source language to help the classification of docu- ments in other label-poor target languages? Existing methods in CLTC use either a bilingual dictionary or a parallel corpus to bridge language barriers and to translate classification models (Xu et al., 2016) or text data(Zhou et al., 2016a). There are limitations and challenges in using ei- ther type of resources. Dictionary-based meth- ods often ignore the dependency of word mean- ing and its context, and cannot leverage domain- specific disambiguation when the dictionary on hand is a general-purpose one. Parallel-corpus based methods, although more effective in de- ploying context (when combined with word em- bedding in particular), often have an issue of do- main mismatch or distribution mismatch if the available source-language training data, the paral- lel corpus (human-aligned or machine-translation induced one) and the target documents of in- terest are not in exactly the same domain and genre(Duh et al., 2011). How …

Content truncated to 500 words