XQA: A Cross-lingual Open-domain Question Answering Dataset

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2358–2368 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2358 XQA: A Cross-lingual Open-domain Question Answering Dataset Jiahua Liu, Yankai Lin, Zhiyuan Liu, Maosong Sun∗ Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab on Intelligent Technology and Systems, Tsinghua University, Beijing, China alphaf52@gmail.com,linyk14@mails.tsinghua.edu.cn {liuzy,sms}@tsinghua.edu.cn Abstract Open-domain question answering (OpenQA) aims to answer questions through text retrieval and reading comprehension. Recently, lots of neural network-based models have been proposed and achieved promising results in OpenQA. However, the success of these mod- els relies on a massive volume of training data (usually in English), which is not available in many other languages, especially for those low-resource languages. Therefore, it is es- sential to investigate cross-lingual OpenQA. In this paper, we construct a novel dataset XQA for cross-lingual OpenQA research. It con- sists of a training set in English as well as development and test sets in eight other lan- guages. Besides, we provide several baseline systems for cross-lingual OpenQA, including two machine translation-based methods and one zero-shot cross-lingual method (multilin- gual BERT). Experimental results show that the multilingual BERT model achieves the best results in almost all target languages, while the performance of cross-lingual OpenQA is still much lower than that of English. Our analysis indicates that the performance of cross-lingual OpenQA is related to not only how similar the target language and English are, but also how difficult the question set of the target language is. The XQA dataset is publicly available at http://github.com/thunlp/XQA. 1 Introduction In recent years, open-domain question answering (OpenQA), which aims to answer open-domain questions with a large-scale text corpus, has at- tracted lots of attention from natural language pro- cessing researchers. Chen et al. (2017) proposed DrQA model, which used a text retriever to obtain relevant documents from Wikipedia, and further applied a trained reading comprehension model ∗Corresponding author: Maosong Sun to extract the answer from the retrieved docu- ments. Moreover, researchers have introduced more sophisticated models, which either aggre- gate all informative evidence (Lin et al., 2018; Wang et al., 2018b) or filter out those noisy re- trieved text (Clark and Gardner, 2018; Choi et al., 2017; Wang et al., 2018a) to better predict the an- swers for open-domain questions. Benefiting from the power of neural networks, these models have achieved remarkable results in OpenQA. How- ever, these neural-based models must be trained with a huge volume of labeled data. Collecting and labeling large-size training data for each lan- guage is often intractable and unrealistic, espe- cially for those low-resource languages. In this case, it is impossible to directly apply existing OpenQA models to many different languages. To address this problem, an alternative approach is to build a cross-lingual OpenQA system. is trained on data in one high-resource sourceIt language such as English, and predicts answers for open-domain questions in other target lan- guages. In fact, cross-lingual OpenQA can be viewed as …

Content truncated to 500 words