Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3604–3622 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3604 Cross-Lingual Text Classification with Minimal Resources by Transferring a Sparse Teacher Giannis Karamanolakis, Daniel Hsu, Luis Gravano Columbia University, New York, NY 10027, USA {gkaraman, djhsu, gravano}@cs.columbia.edu Abstract Cross-lingual text classification alleviates the need for manually labeled documents in a target language by leveraging labeled docu- ments from other languages. Existing ap- proaches for transferring supervision across languages require expensive cross-lingual re- sources, such as parallel corpora, while less ex- pensive cross-lingual representation learning approaches train classifiers without target la- beled documents. In this work, we propose a cross-lingual teacher-student method, CLTS, that generates “weak” supervision in the tar- get language using minimal cross-lingual re- sources, in the form of a small number of word translations. Given a limited translation bud- get, CLTS extracts and transfers only the most important task-specific seed words across lan- guages and initializes a teacher classifier based on the translated seed words. Then, CLTS iter- atively trains a more powerful student that also exploits the context of the seed words in un- labeled target documents and outperforms the teacher. CLTS is simple and surprisingly effec- tive in 18 diverse languages: by transferring just 20 seed words, even a bag-of-words logis- tic regression student outperforms state-of-the- art cross-lingual methods (e.g., based on mul- tilingual BERT). Moreover, CLTS can accom- modate any type of student classifier: leverag- ing a monolingual BERT student leads to fur- ther improvements and outperforms even more expensive approaches by up to 12% in accu- racy. Finally, CLTS addresses emerging tasks in low-resource languages using just a small number of word translations. 1 Introduction The main bottleneck in using supervised learning for multilingual document classification is the high cost of obtaining labeled documents for all of the target languages. To address this issue in a target language LT , we consider a cross-lingual text clas- Teacher Cross-Lingual Transfer of Seed Words Seed Word Extractor Teacher-Student Co-Training “… a wonderful book filled with engaging stories…” ̂W “wonderful” “disappointing” “magnifique” “décevant” POS / NEG Student “… c’est une magnifique histoire que j’ai dévorée…” Source Language Target Language Figure 1: Our cross-lingual teacher-student (CLTS) method trains a student classifier in the target language by transferring weak supervision across languages. sification approach that requires labeled documents only in a source language LS and not in LT . Existing approaches for transferring supervision across languages rely on large parallel corpora or machine translation systems, which are expensive to obtain and are not available for many languages.1 To scale beyond high-resource languages, multi- lingual systems have to reduce the cross-lingual requirements and operate under a limited budget of cross-lingual resources. Such systems typically ignore target-language supervision, and rely on fea- ture representations that bridge languages, such as cross-lingual word embeddings (Ruder et al., 2019) or multilingual transformer models (Wu and Dredze, 2019; Pires et al., 2019). This general ap- proach is less expensive but has a key …

Content truncated to 500 words