University Entrance Examinations as a Benchmark Resource for NLP-based Problem Solving

International Joint Conference on Natural Language Processing, pages 1357–1365, Nagoya, Japan, 14-18 October 2013. University Entrance Examinations as a Benchmark Resource for NLP-based Problem Solving Yusuke Miyao National Institute of Informatics Graduate University for Advanced Studies yusuke@nii.ac.jp Ai Kawazoe National Institute of Informatics zoeai@nii.ac.jp Abstract This paper describes a corpus com- prised of university entrance examina- tions, which is aimed to promote research on NLP-based problem solving. Since en- trance examinations are created for quan- tifying human ability of problem solving, they are a desirable resource for bench- marking NLP-based problem solving sys- tems. However, as entrance examinations involve a variety of subjects and types of questions, in order to pursue focused re- search on specific NLP technologies, it is necessary to break down entire examina- tions into individual NLP subtasks. For this purpose, we provide annotations of question classifications in terms of an- swer types and knowledge types. In this paper, we also describe research issues by referring to results of question classi- fication, and introduce two international shared tasks that employed our resource for developing their evaluation data sets. 1 Introduction This paper introduces natural language corpora whose source texts are taken from university en- trance examinations. This resource has been de- veloped aiming at benchmarking NLP systems for problem solving. In general, entrance examina- tions are mostly described in natural language, and the goal of reading the text is clear, viz., to solve questions. Therefore, this is an ideal resource for evaluating end-to-end NLP systems that read nat- ural language text, perform some information pro- cessing, and output answers. University entrance examinations have several desirable features to be used for benchmarking NLP-based problem solving. They are carefully designed for empirically quantifying a certain ability of high-school-level students. Therefore, it is not a trivial task for NLP systems to solve university entrance examinations. On the other hand, it is guaranteed that required knowledge is fairly restricted, and legitimate solutions always exist. Despite such artificial restrictions, investi- gating the entire process of solving entrance ex- aminations is meaningful, because it is expected to reveal true contributions of current NLP tech- nologies to human-like problem solving tasks. In addition, evaluation results are intuitively under- standable, and can be compared directly with hu- man performance. This provides us with empirical evidence for analyzing the relationships between human intelligence and artificial intelligence. While it is now clear that university entrance ex- aminations are a useful resource for NLP bench- marking, it is also true that they will not be appro- priate for focusing on individual NLP tasks, be- cause they involve a variety of subjects and types of questions. It is almost hopeless to invent a sin- gle clever algorithm that can solve all types of questions. Therefore, it is necessary to break down entire examinations into NLP subtasks that can be investigated solely. For this aim, we annotate clas- sifications of questions, which allow us to isolate specific NLP subtasks for focused research. An important point here is that, question classification …

Content truncated to 500 words