Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation

Proceedings of the 28th International Conference on Computational Linguistics, pages 2710–2721 Barcelona, Spain (Online), December 8-13, 2020 2710 Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation Junhao Liu1,2∗ Linjun Shou3∗ Jian Pei4 Ming Gong3 Min Yang1† Daxin Jiang3† 1Shenzhen Key Laboratory for High Performance Data Mining, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences 2University of Chinese Academy of Sciences 3Microsoft STCA NLP Group, Beijing, China 4School of Computing Science, Simon Fraser University {jh.liu,min.yang}@siat.ac.cn, {lisho,migon,djiang}@microsoft.com, jpei@cs.sfu.ca Abstract Cross-lingual Machine Reading Comprehension (CLMRC) remains a challenging problem due to the lack of large-scale annotated datasets in low-source languages, such as Arabic, Hindi, and Vietnamese. Many previous approaches use translation data by translating from a rich-source language, such as English, to low-source languages as auxiliary supervision. However, how to effectively leverage translation data and reduce the impact of noise introduced by translation remains onerous. In this paper, we tackle this challenge and enhance the cross-lingual transfer- ring performance by a novel augmentation approach named Language Branch Machine Reading Comprehension (LBMRC). A language branch is a group of passages in one single language paired with questions in all target languages. We train multiple machine reading comprehension (MRC) models proficient in individual language based on LBMRC. Then, we devise a multilin- gual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages. Combining the LBMRC and multilingual distillation can be more robust to the data noises, therefore, improving the model’s cross-lingual ability. Mean- while, the produced single multilingual model can apply to all target languages, which saves the cost of training, inference, and maintenance for multiple models. Extensive experiments on two CLMRC benchmarks clearly show the effectiveness of our proposed method. 1 Introduction Machine Reading Comprehension (MRC) is a central task in natural language understanding (NLU) with many applications, such as information retrieval and dialogue generation. Given a query and a text paragraph, MRC extracts the span of the correct answer from the paragraph. Recently, as a series of large- scale annotated datasets become available, such as SQuAD (Rajpurkar et al., 2016) and TriviaQA (Joshi et al., 2017), the performance of MRC systems has been improved dramatically (Xiong et al., 2017; Hu et al., 2017; Yu et al., 2018; Wang et al., 2016; Seo et al., 2016). Nevertheless, those large-scale, high-quality annotated datasets often only exist in rich-resource languages, such as English, French and German. Correspondingly, the improvement of MRC quality can only benefit those rich-source languages. Annotating a large MRC dataset with high quality for every language is very costly and may even be infeasible (He et al., 2017). MRC in low-resource languages still suffers from the lack of large amounts of high-quality training data. Besides, in real business scenarios, it is not practical to train separate MRC models for each language given that there are thousands of languages existed in the world. Thus multi-lingual MRC (single model for multiple languages) is of strong practical value by greatly reducing the model training, serving …

Content truncated to 500 words