Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5933–5940, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5933 Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model Tsung-Yuan Hsu∗ Chi-liang Liu∗ Graduate Institute of Communication Engineering National Taiwan University {sivia89024, liangtaiwan1230, tlkagkb93901106}@gmail.com Hung-yi Lee Abstract Because it is not feasible to collect training data for every language, there is a growing interest in cross-lingual transfer learning. In this paper, we systematically explore zero- shot cross-lingual transfer learning on reading comprehension tasks with a language repre- sentation model pre-trained on multi-lingual corpus. The experimental results show that with pre-trained language representation zero- shot learning is feasible, and translating the source data into the target language is not nec- essary and even degrades the performance. We further explore what does the model learn in zero-shot setting0. 1 Introduction Reading Comprehension (RC) has become a cen- tral task in natural language processing, with great practical value in various industries. In recent years, many large-scale RC datasets in English (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017; Rajpurkar et al., 2018) have nourished the development of numerous powerful and diverse RC models (Seo et al., 2016; Hu et al., 2018; Wang et al., 2017; Clark and Gardner, 2018; Huang et al., 2017). The state-of-the-art model (Devlin et al., 2018) on SQuAD, one of the most widely used RC bench- marks, even surpasses human-level performance. Nonetheless, RC on languages other than English has been limited due to the absence of sufficient training data. Although some efforts have been made to create RC datasets for Chinese (He et al., 2018; Shao et al., 2018) and Korean (Seungy- oung Lim, 2018), it is not feasible to collect RC datasets for every language since annotation ef- forts to collect a new RC dataset are often far from ∗Equal contribution 0All the modifications of existing corpora used in this paper would be released in https://github.com/ntu-spml- lab/artificial-reading-comprehension-datasets trivial. Therefore, the setup of transfer learning, especially zero-shot learning, is of extraordinary importance. Existing methods (Asai et al., 2018) of cross- lingual transfer learning on RC datasets often count on machine translation (MT) to translate data from source language into target language, or vice versa. These methods may not require a well-annotated RC dataset for the target language, whereas a high-quality MT model is needed as a trade-off, which might not be available when it comes to low-resource languages. In this paper, we leverage pre-trained multilin- gual language representation, for example, BERT learned from multilingual un-annotated sentences (multi-BERT), in cross-lingual zero-shot RC. We fine-tune multi-BERT on the training set in source language, then test the model in target language, with a number of combinations of source-target language pair to explore the cross-lingual ability of multi-BERT. Surprisingly, we find that the mod- els have the ability …

Content truncated to 500 words