A Robust Self-Learning Framework for Cross-Lingual Text Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6306–6310, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6306 A Robust Self-Learning Framework for Cross-Lingual Text Classification Xin Dong Rutgers University New Brunswick, NJ, USA xd48@rutgers.edu Gerard de Melo Rutgers University New Brunswick, NJ, USA gdm@demelo.org Abstract Based on massive amounts of data, recent pre- trained contextual representation models have made significant strides in advancing a num- ber of different English NLP tasks. However, for other languages, relevant training data may be lacking, while state-of-the-art deep learn- ing methods are known to be data-hungry. In this paper, we present an elegantly simple ro- bust self-learning framework to include unla- beled non-English samples in the fine-tuning process of pretrained multilingual represen- tation models. We leverage a multilingual model’s own predictions on unlabeled non- English data in order to obtain additional in- formation that can be used during further fine- tuning. Compared with original multilingual models and other cross-lingual classification models, we observe significant gains in effec- tiveness on document and sentiment classifica- tion for a range of diverse languages. 1 Introduction Owing to notable advances in deep learning and representation learning, important progress has been achieved on text classification, read- ing comprehension, and other NLP tasks. Re- cently, pretrained language representations with self-supervised objectives (Peters et al., 2018; De- vlin et al., 2018; Radford et al., 2018) have fur- ther pushed forward the state-of-the-art on many English tasks. While these sorts of deep models can be trained on different languages, deep mod- els typically require substantial amounts of labeled data for the specific domain of data. Unfortunately, the cost of acquiring new custom-built resources for each combination of language and domain is very high, as it typically requires human annotation. Available resources for domain-specific tasks are often imbalanced be- tween different languages. The scarcity of non- English annotated corpora may preclude our abil- ity to train language-specific machine learning models. In contrast, English-language annota- tions are often readily available to train deep mod- els. Although translation can be an option, human translation is very costly and for many language pairs, any available domain-specific parallel cor- pora are too small to train high-quality machine translation systems. Cross-lingual systems rely on training data from one language to train a model that can be ap- plied to other languages (de Melo and Siersdorfer, 2007), alleviating the training bottleneck issues for low-resource languages. This is facilitated by re- cent advances in learning joint multilingual rep- resentations (Lample and Conneau, 2019; Artetxe and Schwenk, 2018; Devlin et al., 2018). In our work, we propose a self-learning frame- work to incorporate the predictions of the multi- lingual BERT model (Devlin et al., 2018) on non- English data into an English training procedure. The initial multilingual BERT model was simul- taneously pretrained on 104 languages, and has shown to perform well for cross-lingual transfer of natural language tasks (Wu and Dredze, …

Content truncated to 500 words