Automatic Question Generation for Vocabulary Assessment

Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 819–826, Vancouver, October 2005. c⃝2005 Association for Computational Linguistics Automatic Question Generation for Vocabulary Assessment Jonathan C. Brown Gwen A. Frishkoff Maxine Eskenazi Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213 USA Learning Research & Development Center University of Pittsburgh Pittsburgh, PA 15260 USA Language Technologies Institute Carnegie Mellon University Pittsburgh, PA 15213 USA jonbrown@cs.cmu.edu gwenf@pitt.edu max@cs.cmu.edu Abstract In the REAP system, users are automati- cally provided with texts to read targeted to their individual reading levels. To find ap- propriate texts, the user’s vocabulary knowledge must be assessed. We describe an approach to automatically generating questions for vocabulary assessment. Tra- ditionally, these assessments have been hand-written. Using data from WordNet, we generate 6 types of vocabulary ques- tions. They can have several forms, includ- ing wordbank and multiple-choice. We present experimental results that suggest that these automatically-generated ques- tions give a measure of vocabulary skill that correlates well with subject perform- ance on independently developed human- written questions. In addition, strong corre- lations with standardized vocabulary tests point to the validity of our approach to automatic assessment of word knowledge. 1 Introduction The REAP system automatically provides users with individualized authentic texts to read. These texts, usually retrieved from the Web, are chosen to satisfy several criteria. First, they are selected to match the reading level of the student (Collins- Thompson and Callan, 2004). They must also have vocabulary terms known to the student. To meet this goal, it is necessary to construct an accurate model of the student’s vocabulary knowledge (Brown and Eskenazi, 2004). Using this model, the system can locate documents that include a given percentage (e.g., 95%) of words that are known to the student. The remaining percentage (e.g. 5%) consists of new words that the student needs to learn. This percentage is controlled so that there is not so much stretch in the document that the stu- dent cannot focus their attention on understanding the new words and the meaning of the text. After reading the text, the student’s understanding of new words is assessed. The student’s responses are used to update the student model, to support re- trieval of furture documents that take into account the changes in student word knowledge. In this paper, we describe our work on automatic generation of vocabulary assessment questions. We also report results from a study that was designed to assess the validity of the generated questions. In addition to the importance of these assessments in the REAP system, tests of word knowledge are central to research on reading and language and are of practical importance for student placement and in enabling teachers to track improvements in word knowledge throughout the school year. Because tests such as these are traditionally hand-written, development is time-consuming and often relies on methods that are informal and subjective. The re- search described here addresses these issues through development of automated, explicit meth- ods for …

Content truncated to 500 words