Automatic and Human Scoring of Word Definition Responses

Proceedings of NAACL HLT 2007, pages 476–483, Rochester, NY, April 2007. c⃝2007 Association for Computational Linguistics Automatic and human scoring of word definition responses Kevyn Collins-Thompson and Jamie Callan Language Technologies Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA, U.S.A. 15213-8213 {kct|callan}@cs.cmu.edu Abstract Assessing learning progress is a critical step in language learning applications and experiments. In word learning, for exam- ple, one important type of assessment is a definition production test, in which sub- jects are asked to produce a short defini- tion of the word being learned. In current practice, each free response is manually scored according to how well its mean- ing matches the target definition. Manual scoring is not only time-consuming, but also limited in its flexibility and ability to detect partial learning effects. This study describes an effective auto- matic method for scoring free responses to definition production tests. The algo- rithm compares the text of the free re- sponse to the text of a reference definition using a statistical model of text semantic similarity that uses Markov chains on a graph of individual word relations. The model can take advantage of both corpus- and knowledge-based resources. Evalu- ated on a new corpus of human-judged free responses, our method achieved sig- nificant improvements over random and cosine baselines in both rank correlation and label error. 1 Introduction Human language technologies are playing an in- creasingly important role in the science and prac- tice of language learning. For example, intelligent Computer Assisted Language Learning (CALL) sys- tems are being developed that can automatically tai- lor lessons and questions to the needs of individual students (Heilman et al., 2006). One critical task that language tutors, word learning experiments, and related applications have in common is assessing the learning progress of the student or experiment sub- ject during the course of the session. When the task is learning new vocabulary, a vari- ety of tests have been developed to measure word learning progress. Some tests, such as multiple- choice selection of a correct synonym or cloze com- pletion, are relatively passive. In production tests, on the other hand, students are asked to write or say a short phrase or sentence that uses the word being learned, called the target word, in a specified way. In one important type of production test, called a definition production test, the subject is asked to de- scribe the meaning of the target word, as they under- stand it at that point in the session. The use of such tests has typically required a teacher or researcher to manually score each response by judging its sim- ilarity in meaning to the reference definition of the target word. The resulting scores can then be used to analyze how a person’s learning of the word re- sponded to different stimuli, such as seeing the word used in context. A sample target word and its ref- erence definition, along with examples of human- judged responses, are given in Sections 3.3 and 4.1. However, manual …

Content truncated to 500 words