Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3770–3785 August 1–6, 2021. ©2021 Association for Computational Linguistics 3770 Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference Hai Hu†÷× He Zhou† Zuoyu Tian† Yiwen Zhang† Yina Ma‡ Yanting Li◁ Yixin Nie○ Kyle Richardson◇ †Indiana University Bloomington ‡Brigham Young University ÷×Shanghai Jiaotong Univ. ◁Northwestern University ○UNC Chapel Hill ◇Allen Institute for AI hu.hai@outlook.com; {hzh1,zuoytian,yiwezhan,yinama}@iu.edu; yanting.li@northwestern.edu; yixin1@cs.unc.edu; kyler@allenai.org Abstract Multilingual transformers (XLM, mT5) have been shown to have remarkable transfer skills in zero-shot settings. Most transfer studies, however, rely on automatically translated re- sources (XNLI, XQuAD), making it hard to discern the particular linguistic knowledge that is being transferred, and the role of expert an- notated monolingual datasets when develop- ing task-specific models. We investigate the cross-lingual transfer abilities of XLM-R for Chinese and English natural language infer- ence (NLI), with a focus on the recent large- scale Chinese dataset OCNLI. To better un- derstand linguistic transfer, we created 4 cat- egories of challenge and adversarial tasks (to- taling 17 new datasets1) for Chinese that build on several well-known resources for English (e.g., HANS, NLI stress-tests). We find that cross-lingual models trained on English NLI do transfer well across our Chinese tasks (e.g., in 3/4 of our challenge categories, they per- form as well/better than the best monolingual models, even on 3/5 uniquely Chinese lin- guistic phenomena such as idioms, pro drop). These results, however, come with important caveats: cross-lingual models often perform best when trained on a mixture of English and high-quality monolingual NLI data (OCNLI), and are often hindered by automatically trans- lated resources (XNLI-zh). For many phenom- ena, all models continue to struggle, highlight- ing the need for our new diagnostics to help benchmark Chinese and cross-lingual models. 1 Introduction Recent pre-trained multilingual transformer mod- els, such as XLM(-R) (Conneau and Lample, 2019; Conneau et al., 2020), mT5 (Xue et al., 2020) and others (Liu et al., 2020; Lewis et al., 2020) have 1All new datasets/code are released at https://github.com/ huhailinguist/ChineseNLIProbing. been shown to be successful in NLP tasks for sev- eral non-English languages (Khashabi et al., 2020; Choi et al., 2021), as well as in multilingual bench- marks (Devlin et al., 2019; Conneau et al., 2020; Xue et al., 2020; Artetxe et al., 2020). A particular appeal is that they can be used for cross-lingual and zero-shot transfer. That is, after pre-training on a raw, unaligned corpus consisting of text from many languages, models can be subsequently fine-tuned on a particular task in a resource-rich language (e.g., English) and directly applied to the same task in other languages without requiring any additional language-specific training. Given this recent progress, a natural question arises: does it make sense to invest in large-scale task-specific dataset construction for low-resourced languages, or does cross-lingual transfer alone suf- fice for many languages and tasks? A closely re- lated question is: how well do multilingual mod- els transfer across specific linguistic and language- specific phenomena? While there has been …

Content truncated to 500 words