Dataset for the First Evaluation on Chinese Machine Reading Comprehension

Dataset for the First Evaluation on Chinese Machine Reading Comprehension Yiming Cui†, Ting Liu‡, Zhipeng Chen†, Wentao Ma†, Shijin Wang† and Guoping Hu† †Joint Laboratory of HIT and iFLYTEK, iFLYTEK Research, Beijing, China ‡Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin, China †{ymcui,zpchen,wtma,sjwang3,gphu}@iflytek.com ‡tliu@ir.hit.edu.cn Abstract Machine Reading Comprehension (MRC) has become enormously popular recently and has attracted a lot of attention. However, existing reading comprehension datasets are mostly in English. To add diversity in reading comprehension datasets, in this paper we propose a new Chinese reading comprehension dataset for accelerating related research in the community. The proposed dataset contains two different types: cloze-style reading comprehension and user query reading comprehension, associated with large-scale training data as well as human-annotated validation and hidden test set. Along with this dataset, we also hosted the first Evaluation on Chinese Machine Reading Comprehension (CMRC-2017) and successfully attracted tens of participants, which suggest the potential impact of this dataset. Keywords: Chinese Reading Comprehension, Question Answering, Evaluation 1. Introduction Machine Reading Comprehension (MRC) has become enormously popular in recent research, which aims to teach the machine to comprehend human languages and answer the questions based on the reading materials. Among var- ious reading comprehension tasks, the cloze-style reaing comprehension is relatively easy to follow due to its sim- plicity in definition, which requires the model to fill an exact word into the query to form a coherent sentence ac- cording to the document material. Several cloze-style read- ing comprehension datasets are publicly available, such as CNN/Daily Mail (Hermann et al., 2015), Children’s Book Test (Hill et al., 2015), People Daily and Children’s Fairy Tale (Cui et al., 2016). In this paper, we provide a new Chinese reading compre- hension dataset1, which has the following features • We provide a large-scale automatically generated Chinese cloze-style reading comprehension dataset, which is gathered from children’s reading material. • Despite the automatic generation of training data, our evaluation datasets (validation and test) are annotated manually, which is different from previous works. • To add more diversity and for further investigation on transfer learning, we also provide another evaluation datasets which is also annotated by human, but the query is more natural than the cloze type. We also host the 1st Evaluation on Chinese Machine Read- ing Comprehension (CMRC2017), which has attracted over 30 participants and finally there were 17 participants sub- mitted their evaluation systems for testing their reading comprehension models on our newly developed dataset, suggesting its potential impact. We hope the release of the 1CMRC 2017 Public Datasets: https://github.com/ ymcui/cmrc2017. dataset to the public will accelerate the progress of Chi- nese research community on machine reading comprehen- sion field. We also provide four official baselines for the evaluations, including two traditional baselines and two neural base- lines. In this paper, we adopt two widely used neural read- ing comprehension model: AS Reader (Kadlec et al., 2016) and AoA Reader (Cui et al., 2017). The rest of the paper will be organized as follows. …

Content truncated to 500 words