Using Lexical Semantic Techniques to Classify Free-Responses

Using Lexical Semantic Techniques to Classify Free-Responses Jill Burstein Educational Testing Service - 11R Princeton, New Jersey 08541 jburstein@ets.org Randy Kaplan Educational Testing Service - 17R Princeton, New Jersey 08541 rkaplan @ets.org Susanne Wolff Educational Testing Service - 17R Princeton, New Jersey 08541 swolff@ets.org Chi Lu Educational Testing Service - 17R Princeton, New Jersey 08541 clu@ets.org Abstract This paper discusses a case study in which lexical semantic techniques were used to implement a prototype scoring system for short-answer, free-responses to test questions. Scoring, as it is discussed in this paper, is a kind of clasgification problem. Responses are automatically scored by being assigned appropriate classifications. The ultimate goal is to develop a scoring system which can reliably analyze response content. For this study, a domain-specific, concept-based lexicon, and a concept grammar were built to represent the response set, using 200 of 378 responses from the original data set. The lexicon is built, from individual words, and 2-word and 3-word terms from the training data. The lexicon is best characterized by Bergler's (1995) layered lexicon. Concept grammar rules are built by mapping concepts from the lexicon onto the concept-structure patterns present in a set of training responses. Previous attempts to score these responses using lexically-based statistical techniques and structure-independent content grammars were not reliable (Burstein and Kaplan (1995)). The results discussed in this paper illustrate the reliability of the lexical semantic methods used in the study. 20 ! 1. Introduction There is a movement in testing to augment the conventional multiple-choice items (i.e., test questions) with short-answer free-response items. Due to the large volume of tests administered yearly by Educational Testing Service (ETS), hand-scoring of these tests with these types of items is costly and time-consuming for practical testing programs. ETS is currently working on natural language understanding systems which could be used for computer-assisted scoring of short-answer free- responses (see Kaplan and Bennett (1994) and Burstein and Kaplan (1995))) The overall goal of our current research is to develop a scoring system that can handle short-answer free-response items. Such a scoring system has to be able to identify the relevant content of a response and assign it to an appropriate content category. Another consideration in the development of a scoring system is that the data sets that are available to us are relatively small, and the responses in these data sets lack lexico-~syntactic patterning. The items which we work with are either experimental, or have been administered as paper-and-pencil exams. In the former case, there is a limited subject pool, and in the latter case, we rely on what has been put into electronic form. The response sets typically range from 300-700 responses which we have to use for training and testing. This is quite a different scenario from natural language understanding systems which can be designed using large corpora from full text sources, such as the AP News and the Wall Street Journal. This paper discusses a case study that examined how lexical semantic techniques could be used to …

Content truncated to 500 words