A Sentence Cloze Dataset for Chinese Machine Reading Comprehension

Proceedings of the 28th International Conference on Computational Linguistics, pages 6717–6723 Barcelona, Spain (Online), December 8-13, 2020 6717 A Sentence Cloze Dataset for Chinese Machine Reading Comprehension Yiming Cui†‡, Ting Liu†, Ziqing Yang‡, Zhipeng Chen‡, Wentao Ma‡, Wanxiang Che†, Shijin Wang‡§, Guoping Hu‡ †Research Center for SCIR, Harbin Institute of Technology, Harbin, China ‡State Key Laboratory of Cognitive Intelligence, iFLYTEK Research, China §iFLYTEK AI Research (Hebei), Langfang, China †{ymcui,tliu,car}@ir.hit.edu.cn ‡§{ymcui,zqyang5,zpchen,wtma,sjwang3,gphu}@iflytek.com Abstract Owing to the continuous efforts by the Chinese NLP community, more and more Chinese ma- chine reading comprehension datasets become available. To add diversity in this area, in this paper, we propose a new task called Sentence Cloze-style Machine Reading Comprehension (SC-MRC). The proposed task aims to fill the right candidate sentence into the passage that has several blanks. We built a Chinese dataset called CMRC 2019 to evaluate the difficulty of the SC-MRC task. Moreover, to add more difficulties, we also made fake candidates that are similar to the correct ones, which requires the machine to judge their correctness in the con- text. The proposed dataset contains over 100K blanks (questions) within over 10K passages, which was originated from Chinese narrative stories. To evaluate the dataset, we implement several baseline systems based on the pre-trained models, and the results show that the state- of-the-art model still underperforms human performance by a large margin. We release the dataset and baseline system to further facilitate our community. Resources available through https://github.com/ymcui/cmrc2019 1 Introduction Machine Reading Comprehension (MRC) is a task to comprehend given articles and answer the questions based on them, which is an important ability for artificial intelligence. The recent MRC research was originated from the cloze-style reading comprehension (Hermann et al., 2015; Hill et al., 2015; Cui et al., 2016), which requires to fill in the blank with a word or named entity, and following works on these datasets have laid the foundations of this research (Kadlec et al., 2016; Cui et al., 2017; Dhingra et al., 2017). Later on, SQuAD (Rajpurkar et al., 2016) was proposed, and the answer transformed from a single word to a span, which has become a representative span-extraction dataset and massive neural network approaches (Wang and Jiang, 2016; Xiong et al., 2016; Wang et al., 2017; Hu et al., 2018; Wang et al., 2018; Yu et al., 2018) have been proposed which further accelerated the MRC research. Besides the MRC in English text, we have also seen rapid progress on Chinese MRC research. Cui et al. (2016) proposed the first Chinese cloze-style reading comprehension dataset: People Daily & Children’s Fairy Tale (PD&CFT). Later, Cui et al. (2018) proposed another dataset for CMRC 2017, which is gathered from children’s reading books, consisting of both cloze and natural questions. He et al. (2018) proposed a large-scale open-domain Chinese reading comprehension dataset (DuReader), which consists of 200k queries annotated from the user query logs on the search engine. In span-extraction MRC, Cui et al. (2019b) proposed CMRC 2018 dataset for Simplified Chinese, and Shao et …

Content truncated to 500 words