Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension

Investigating Prior Knowledge for Challenging Chinese Machine Reading Comprehension Kai Sun1∗ Dian Yu2 Dong Yu2 Claire Cardie1 1Cornell University, Ithaca, NY 2Tencent AI Lab, Bellevue, WA ks985@cornell.edu,{yudian, dyu}@tencent.com,cardie@cs.cornell.edu Abstract Machine reading comprehension tasks require a machine reader to answer questions relevant to the given document. In this paper, wepresent the first free-form multiple-Choice Chinese machine reading Comprehension dataset (C3), containing 13,369 documents (dialogues or more formally written mixed-genre texts) and their associated 19,577 multiple-choice free-form questions collected from Chinese- as-a-second-language examinations. We present a comprehensive analysis of the prior knowledge (i.e., linguistic, domain- specific, and general world knowledge) needed for these real-world problems. We implement rule-based and popular neural methods and find that there is still a significant performance gap between the best performing model (68.5%) and human readers (96.0%), especiallyon problems that require prior knowledge.Wefur- ther study the effects of distractor plausibility and data augmentation based on translated relevant datasets for English on model perfor- mance. We expect C3 to present great chal- lenges to existing systems as answering 86.8% of questions requires both knowledge within and beyond the accompanying document, and we hope that C3 can serve as a platform to study how to leverage various kinds of prior knowledge to better understand a given writ- ten or orally oriented text. C3 is available at https://dataset.org/c3/. 1 Introduction ‘‘Language is, at best, a means of directing others to construct similar-thoughts from their own prior knowledge.’’ Adams and Bruce (1982) ∗Part of this work was conducted when K. S. was an intern at the Tencent AI Lab, Bellevue, WA. Machine reading comprehension (MRC) tasks have attracted substantial attention from both academia and industry. These tasks require a machine reader to answer questions relevant to a given document provided as input (Poon et al., 2010; Richardson et al., 2013). In this paper, we focus on free-form multiple-choice MRC tasks—given a document, select the correct ans- wer option from all options associated with a free- form question, which is not limited to a single question type such as cloze-style questions formed by removing a span or a sentence in a text (Hill et al., 2016; Bajgar et al., 2016; Mostafazadeh et al., 2016; Xie et al., 2018; Zheng et al., 2019) or close-ended questions that can be answered with a minimal answer (e.g., yes or no; Clark et al., 2019). Researchers have developed a variety of free- form multiple-choice MRC datasets that contain a significant percentage of questions focusing on the implicitly expressed facts, events, opinions, or emotions in the given text (Richardson et al., 2013; Lai et al., 2017; Ostermann et al., 2018; Khashabi et al., 2018; Sun et al., 2019a). Generally, we re- quire the integration of our own prior knowledge and the information presented in the given text to answer these questions, posing new challenges for MRC systems. However, until recently, progress in the development of techniques for addressing this kind of MRC task for Chinese has lagged behind their English counterparts. A primary rea- son is that …

Content truncated to 500 words