Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1125–1136, Osaka, Japan, December 11-17 2016. Generating Questions and Multiple-Choice Answers using Semantic Analysis of Texts Jun Araki, Dheeraj Rajagopal, Sreecharan Sankaranarayanan, Susan Holm, Yukari Yamakawa, Teruko Mitamura Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 15213, USA junaraki@cs.cmu.edu, dheeraj@cs.cmu.edu, sree@cmu.edu, sh4s@andrew.cmu.edu, yukariy@andrew.cmu.edu, teruko@cs.cmu.edu Abstract We present a novel approach to automated question generation that improves upon prior work both from a technology perspective and from an assessment perspective. Our system is aimed at engaging language learners by generating multiple-choice questions which utilize specific in- ference steps over multiple sentences, namely coreference resolution and paraphrase detection. The system also generates correct answers and semantically-motivated phrase-level distractors as answer choices. Evaluation by human annotators indicates that our approach requires a larger number of inference steps, which necessitate deeper semantic understanding of texts than a tra- ditional single-sentence approach. 1 Introduction Exam questions are an indispensable tool for teachers to assess their students’ understanding of ma- terial. Thus, automatic question generation from text is a key natural language processing technology to aid teachers in examining learners’ reading comprehension. Past studies in education showed that higher-level questions, in contrast to simple factoid questions, have more educational benefits for read- ing comprehension (Anderson and Biddle, 1975; Andre, 1979; Hamaker, 1986). However, most of existing approaches to question generation have focused on generating questions from a single sentence, relying heavily on syntax and shallow semantics with an emphasis on grammaticality (Mitkov and Ha, 2003; Chen et al., 2009; Heilman and Smith, 2010; Curto et al., 2011; Becker et al., 2012; Lindberg et al., 2013; Mazidi and Nielsen, 2014). A problem with this approach is that the majority of questions generated from single sentences tend to be too specific and low-level to properly measure learners’ un- derstandings of the overall contents of text. In other words, what is assessed by such question generation systems ends up essentially being the ability to compare sentences, just requiring learners to find a single sentence that has almost the same surface form as a given interrogative sentence. Results of simple sen- tence comparisons do little to contribute towards the goal of assessing learners’ reading comprehension. In this work, we propose a question generation approach that engages learners through the use of specific inference steps over multiple sentences, namely coreference resolution and paraphrase detection, requiring more semantic understanding of text. We primarily use event and entity coreference as a source of producing questions from multiple sentences. Grounded by the past studies in education, we believe that such high-level questions are more sophisticated and educationally valuable for testing reading comprehension than questions generated by the traditional single-sentence approaches. Our question generation strategy is novel in two ways. First, it employs event and entity coreference between antecedents and referring mentions spanning multiple sentences. This requires learners to resolve the coreference and understand the contents of the text semantically. Second, it makes use of paraphrases when generating questions. The resulting …

Content truncated to 500 words