Linguistic Appropriateness and Pedagogic Usefulness of Reading Comprehension Questions

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1753–1762 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1753 Linguistic Appropriateness and Pedagogic Usefulness of Reading Comprehension Questions Andrea Horbach∗, Itziar Aldabe♦, Marie Bexte∗, Oier Lopez de Lacalle♦, Montse Maritxalar♦ ∗ Language Technology Lab, University Duisburg-Essen, Germany ♦ University of the Basque Country (UPV/EHU), Spain ∗ {andrea.horbach,marie.bexte}@uni-due.de ♦ {itziar.aldabe,oier.lopezdelacalle,montse.maritxalar}@ehu.eus Abstract Automatic generation of reading comprehension questions is a topic receiving growing interest in the NLP community, but there is currently no consensus on evaluation metrics and many approaches focus on linguistic quality only while ignoring the pedagogic value and appropriateness of questions. This paper overcomes such weaknesses by a new evaluation scheme where questions from the questionnaire are structured in a hierarchical way to avoid confronting human annotators with evaluation measures that do not make sense for a certain question. We show through an annotation study that our scheme can be applied, but that expert annotators with some level of expertise are needed. We also created and evaluated two new evaluation data sets from the biology domain for Basque and German, composed of questions written by people with an educational background, which will be publicly released. Results show that manually generated questions are in general both of higher linguistic as well as pedagogic quality and that among the human generated questions, teacher-generated ones tend to be most useful. Keywords: question generation, reading comprehension, evaluation guidelines 1. Introduction/Motivation The automatic generation of reading comprehension ques- tions for educational purposes has been a topic receiving considerable attention in the NLP community (Du et al., 2017; Heilman, 2011; Mazidi and Nielsen, 2014). A vari- ety of approaches has been proposed and evaluated. How- ever, a recent review of such systems and their evaluations, e.g., Amidei et al. (2018) showed that many of these eval- uation measures focus on linguistic quality of the produced questions only and often completely ignore the educational appropriateness of a question. A question might be linguis- tically correct and well-formed, but may not be helpful in pedagogical terms. Consider Figure 1. showing an example from the SQuAD data set (Rajpurkar et al., 2016), one of the most frequently used data sets for training automatic current question gen- eration systems. One can see that the questions for this passage are factoid questions whose answer can be found directly in one place in the text. While this was a design decision for the data set, originally created for the task of machine reading, this can be problematic when used to au- tomatically generate questions in an educational scenario, where a teacher might feel the need to go beyond such rel- atively simple questions. This might also be the reason why evaluation measures for such questions target linguis- tic quality only. In this paper we provide a novel annotation scheme for evaluating linguistic appropriateness and pedagogic useful- ness of reading comprehension questions that should make it easy for the evaluators to follow the …

Content truncated to 500 words