English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 557–575 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 557 English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too Jason Phang1,⇤Iacer Calixto1,2,⇤ Phu Mon Htut1 Yada Pruksachatkun1 Haokun Liu1 Clara Vania1 Katharina Kann3 Samuel R. Bowman1 1New York University 2ILLC, University of Amsterdam 3University of Colorado Boulder {jasonphang,iacer.calixto,bowman}@nyu.edu Abstract Intermediate-task training—fine-tuning a pre- trained model on an intermediate task before fine-tuning again on the target task—often improves model performance substantially on language understanding tasks in monolingual English settings. We investigate whether En- glish intermediate-task training is still helpful on non-English target tasks. Using nine in- termediate language-understanding tasks, we evaluate intermediate-task transfer in a zero- shot cross-lingual setting on the XTREME benchmark. We see large improvements from intermediate training on the BUCC and Tatoeba sentence retrieval tasks and moder- ate improvements on question-answering tar- get tasks. MNLI, SQuAD and HellaSwag achieve the best overall results as interme- diate tasks, while multi-task intermediate of- fers small additional improvements. Using our best intermediate-task models for each tar- get task, we obtain a 5.4 point improvement over XLM-R Large on the XTREME bench- mark, setting the state of the art1 as of June 2020. We also investigate continuing multi- lingual MLM during intermediate-task train- ing and using machine-translated intermediate- task data, but neither consistently outperforms simply performing English intermediate-task training. 1 Introduction Zero-shot cross-lingual transfer involves training a model on task data in one set of languages (or language pairs, in the case of translation) and eval- uating the model on the same task in unseen lan- guages (or pairs). In the context of natural language understanding tasks, this is generally done using a pretrained multilingual language-encoding model ⇤⇤Equal contribution. 1The state of art on XTREME at the time of final publi- cation in September 2020 is held by Fang et al. (2020), who introduce an orthogonal method. such as mBERT (Devlin et al., 2019a), XLM (Con- neau and Lample, 2019) or XLM-R (Conneau et al., 2020) that has been pretrained with a masked lan- guage modeling (MLM) objective on large corpora of multilingual data, fine-tune it on task data in one language, and evaluate the tuned model on the same task in other languages. Intermediate-task training (STILTs; Phang et al., 2018) consists of fine-tuning a pretrained model on a data-rich intermediate task, before fine-tuning a second time on the target task. Despite its simplic- ity, this two-phase training setup has been shown to be helpful across a range of Transformer models and target tasks (Wang et al., 2019a; Pruksachatkun et al., 2020), at least within English settings. In this work, we propose to use intermediate training on English tasks to improve zero-shot cross-lingual transfer performance. Starting with a pretrained multilingual language encoder, we per- form intermediate-task training on one or more English tasks, then fine-tune on the target task in English, and finally evaluate zero-shot …

Content truncated to 500 words