Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 806–817 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1075 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 806–817 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1075 Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability Saku Sugawara♠, Yusuke Kido♠, Hikaru Yokono♣, and Akiko Aizawa♦♠ ♠The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan ♣Fujitsu Laboratories Ltd., 4-1-1 Kamikodanaka, Nakahara-ku, Kawasaki, Japan ♦Natural Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, Japan sakus@is.s.u-tokyo.ac.jp mail@yusuk.eki.do yokono.hikaru@jp.fujitsu.com aizawa@nii.ac.jp Abstract Knowing the quality of reading compre- hension (RC) datasets is important for the development of natural-language un- derstanding systems. In this study, two classes of metrics were adopted for eval- uating RC datasets: prerequisite skills and readability. We applied these classes to six existing datasets, including MCTest and SQuAD, and highlighted the characteris- tics of the datasets according to each met- ric and the correlation between the two classes. Our dataset analysis suggests that the readability of RC datasets does not di- rectly affect the question difficulty and that it is possible to create an RC dataset that is easy to read but difficult to answer. 1 Introduction A major goal of natural language processing (NLP) is to develop agents that can understand natural language. Such an ability can be tested with a reading comprehension (RC) task that re- quires the agent to read open-domain documents and answer questions about them. Constructing systems with RC competence is challenging be- cause RC comprises multiple processes includ- ing parsing, understanding cohesion, and infer- ence with linguistic and general knowledge. Clarifying what a system achieves is important in the development of RC systems. To achieve robust improvement, systems should be measured according to a variety of metrics beyond simple accuracy. However, a current problem is that most RC datasets are presented only with superfi- cial categories, such as question types (e.g., what, where, and who) and answer types (e.g., numeric, location, and person). In addition, Chen et al. (2016) noted that some questions in datasets may not be suited to the testing of RC systems. In such ID: SQuAD, United Methodist Church Context: The United Methodist Church (UMC) prac- tices infant and adult baptism. Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question: What are members who have been baptized as an infant or child but who have not subsequently pro- fessed their own faith? Answer: Baptized Members ID: MCTest, mc160.dev.8 Context: Sara wanted to play on a baseball team. She had never tried to swing a bat and hit a baseball before. Her Dad gave her a bat and together they went to the park to practice. Question: Why was Sara practicing? Answer: She wanted to play on a team Figure 1: Examples of RC questions from SQuAD (Rajpurkar et al., 2016) …

Content truncated to 500 words