XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6008–6018, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6008 XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation Yaobo Liang, �Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, Ming Zhou {yalia,nanduan,yegong,t-niwu,v-fengu,v-weqi,migon,lisho,djiang,gucao,xiafan,bzhang,rahul.agrawal,edwac, sinwei,tbharti,yiqia,jiuche,winniew,shuguanl,fanyang,Campos.Daniel, ranganm,mingzhou}@microsoft.com Abstract In this paper, we introduce XGLUE, a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora and evaluate their performance across a diverse set of cross-lingual tasks. Comparing to GLUE (Wang et al., 2019), which is labeled in En- glish for natural language understanding tasks only, XGLUE has two main advantages: (1) it provides 11 diversified tasks that cover both natural language understanding and generation scenarios; (2) for each task, it provides labeled data in multiple languages. We extend a re- cent cross-lingual pre-trained model Unicoder (Huang et al., 2019) to cover both understand- ing and generation tasks, which is evaluated on XGLUE as a strong baseline. We also evalu- ate the base versions (12-layer) of Multilingual BERT, XLM and XLM-R for comparison. 1 1 Introduction Pre-training + Fine-tuning has become a new NLP paradigm, where the general knowledge are firstly learnt from large-scale corpus by self-supervised learning and then transferred to downstream tasks by task-specific fine-tuning. Three different types of pre-trained models are explored recently, includ- ing monolingual pre-trained models (Radford et al., 2018; Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019b; Dong et al., 2019; Lewis et al., 2019a), multilingual and cross-lingual pre-trained models (Devlin et al., 2019; Conneau and Lample, 2019; Huang et al., 2019; Conneau et al., 2019) and mul- timodal pre-trained models (Lu et al., 2019; Li et al., 2020; Chen et al., 2019; Zhou et al., 2020). In this paper, we focus on the cross-lingual pre- trained models, due to their importance to alle- viating the low-resource issue among languages, 1The dataset is available at https://microsoft. github.io/XGLUE/, The code and model is available at https://github.com/microsoft/Unicoder where an NLP task often has rich training data in one language (such as English) but has few or no training data in other languages (such as French and German). In order to further advance the de- velopment of cross-lingual pre-trained models for various downstream tasks in different languages, this paper introduces XGLUE, a new benchmark dataset that can be used to: (i) train large-scale cross-lingual pre-trained models using multilingual and bilingual corpora, (ii) evaluate generalization capabilities of the cross-lingual pre-trained models across a diverse set of cross-lingual tasks. The contribution of XGLUE is two-fold. First, it provides 11 diversified cross-lingual tasks cov- ering both understanding and generation scenarios. XTREME (Hu et al., 2020) is a concurrent work of XGLUE. But it includes cross-lingual under- standing tasks only. Besides, XGLUE …

Content truncated to 500 words