BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2452–2462, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 2452 BiPaR: A Bilingual Parallel Dataset for Multilingual and Cross-lingual Reading Comprehension on Novels Yimin Jing , Deyi Xiong∗and Yan Zhen School of Computer Science and Technology, Soochow University, Suzhou, China {yymmjing,corezhen}@gmail.com dyxiong@suda.edu.cn Abstract This paper presents BiPaR, a bilingual parallel novel-style machine reading comprehension (MRC) dataset, developed to support multi- lingual and cross-lingual reading comprehen- sion. The biggest difference between BiPaR and existing reading comprehension datasets is that each triple (Passage, Question, Answer) in BiPaR is written parallelly in two languages. We collect 3,667 bilingual parallel paragraphs from Chinese and English novels, from which we construct 14,668 parallel question-answer pairs via crowdsourced workers following a strict quality control procedure. We analyze BiPaR in depth and find that BiPaR offers good diversification in prefixes of questions, answer types and relationships between ques- tions and passages. We also observe that answering questions of novels requires read- ing comprehension skills of coreference res- olution, multi-sentence reasoning, and under- standing of implicit causality, etc. With Bi- PaR, we build monolingual, multilingual, and cross-lingual MRC baseline models. Even for the relatively simple monolingual MRC on this dataset, experiments show that a strong BERT baseline is over 30 points behind human in terms of both EM and F1 score, indicating that BiPaR provides a challenging testbed for monolingual, multilingual and cross-lingual MRC on novels. The dataset is available at https://multinlp.github.io/BiPaR/. 1 Introduction Machine reading comprehension is to evaluate how well computer systems understand natural language texts, where machines read a given text passage and answer questions about the passage. It has been regarded as a crucial technology for many applications such as question answering, di- alogue systems (Nguyen et al., 2016; Chen et al., ∗Corresponding author ��������������������������� ������������������������� ������������������������ ������������������������������������������������������� ������������������������������������������������������������ ������������������������������������������������������������� ��������������������������������������������������������� ������������������������������������������������������� ������������������������������� ������������������������������������������� ��������������������� ������������������������������������������������� ��������������� ������������������������������� ����������������������������������������������� ����������������� ����������������� ��������������� ��������������������������������� ����������������������������������������������������� ������������������������������������������������� ����������������������������� ���������������� Figure 1: Illustration of BiPaR with the monolingual, multilingual and cross-lingual MRC on the dataset. 2017; Liu et al., 2018; Wang et al., 2018) and so on. In order to enable machine to understand texts, large-scale reading comprehension datasets have been developed, such as CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016), MS MACRO (Nguyen et al., 2016), hot- potQA (Yang et al., 2018), CoQA (Reddy et al., 2019), etc. The majority of such datasets, unfortunately, are only for monolingual text understanding. To the best of our knowledge, there is no publicly available bilingual parallel reading comprehension dataset, which is exactly what BiPaR is mainly developed for, as illustrated in Figure 1. BiPaR 2453 Dataset Language Parallel Answer Type Domain CNN/DM (Hermann et al., 2015) HLF-RC (Cui et al., 2016) SQuAD (Rajpurkar et al., 2016) CMRC2018 (Cui et al., 2018) MS MACRO (Nguyen et al., 2016) DuReader (He et al., 2018) EN ZH EN ZH EN ZH ××××× …

Content truncated to 500 words