Generating Commonsense Explanation by Extracting Bridge Concepts from Reasoning Paths

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 12–17 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 12 Can Monolingual Pretrained Models Help Cross-Lingual Classification? Zewen Chi†∗, Li Dong‡, Furu Wei‡, Xian-Ling Mao†, Heyan Huang† †Beijing Institute of Technology ‡Microsoft Research {czw,maoxl,hhy63}@bit.edu.cn {lidong1,fuwei}@microsoft.com Abstract Multilingual pretrained language models (such as multilingual BERT) have achieved impres- sive results for cross-lingual transfer. How- ever, due to the constant model capacity, mul- tilingual pre-training usually lags behind the monolingual competitors. In this work, we present two approaches to improve zero-shot cross-lingual classification, by transferring the knowledge from monolingual pretrained mod- els to multilingual ones. Experimental results on two cross-lingual classification benchmarks show that our methods outperform vanilla multilingual fine-tuning. 1 Introduction Supervised text classification heavily relies on manually annotated training data, while the data are usually only available in rich-resource lan- guages, such as English. It requires great effort to make the resources available in other languages. Various methods have been proposed to build cross-lingual classification models by exploiting machine translation systems (Xu and Yang, 2017; Chen et al., 2018; Conneau et al., 2018), and learning multilingual embeddings (Conneau et al., 2018; Yu et al., 2018; Artetxe and Schwenk, 2019; Eisenschlos et al., 2019). Recently, multilingual pretrained language models have shown surprising cross-lingual effec- tiveness on a wide range of downstream tasks (De- vlin et al., 2019; Conneau and Lample, 2019; Con- neau et al., 2020; Chi et al., 2020a,b). Even with- out using any parallel corpora, the pretrained mod- els can still perform zero-shot cross-lingual classi- fication (Pires et al., 2019; Wu and Dredze, 2019; Keung et al., 2019). That is, these models can be fine-tuned in a source language, and then di- rectly evaluated in other target languages. Despite ∗Contribution during internship at Microsoft Research. the effectiveness of cross-lingual transfer, the mul- tilingual pretrained language models have their own drawbacks. Due to the constant number of model parameters, the model capacity of the rich- resource languages decreases if we adds languages for pre-training. The curse of multilinguality re- sults in that the multilingual models usually per- form worse than their monolingual competitors on downstream tasks (Arivazhagan et al., 2019; Con- neau et al., 2020). The observations motivate us to leverage monolingual pretrained models to im- prove multilingual models for cross-lingual classi- fication. In this paper, we propose a multilingual fine- tuning method (MONOX) based on the teacher- student framework, where a multilingual student model learns end task skills from a monolingual teacher. Intuitively, monolingual pretrained mod- els are used to provide supervision of downstream tasks, while multilingual models are employed for knowledge transfer across languages. We con- duct experiments on two widely used cross-lingual classification datasets, where our methods outper- form baseline models on zero-shot cross-lingual classification. Moreover, we show that the mono- lingual teacher model can help the student mul- tilingual model for both the source language and target languages, …

Content truncated to 500 words