Towards Automatic Scoring of Non-Native Spontaneous Speech

Proceedings of the Human Language Technology Conference of the North American Chapter of the ACL, pages 216–223, New York, June 2006. c⃝2006 Association for Computational Linguistics Towards Automatic Scoring of Non-Native Spontaneous Speech Klaus Zechner and Isaac I. Bejar Educational Testing Service Princeton, NJ, USA (kzechner,ibejar)@ets.org Abstract This paper investigates the feasibility of automated scoring of spoken English proficiency of non-native speakers. Unlike existing automated assessments of spoken English, our data consists of spontaneous spoken responses to complex test items. We perform both a quantitative and a qualitative analysis of these features using two different machine learning approaches. (1) We use support vector machines to produce a score and evaluate it with respect to a mode baseline and to human rater agreement. We find that scoring based on support vector machines yields accuracies approaching inter-rater agreement in some cases. (2) We use classification and regression trees to understand the role of different features and feature classes in the characterization of speaking proficiency by human scorers. Our analysis shows that across all the test items most or all the feature classes are used in the nodes of the trees suggesting that the scores are, appropriately, a combination of multiple components of speaking proficiency. Future research will concentrate on extending the set of features and introducing new feature classes to arrive at a scoring model that comprises additional relevant aspects of speaking proficiency. 1 Introduction While automated scoring of open-ended written discourse has been approached by several groups recently (Rudner & Gagne, 2001; Sher- mis & Burstein, 2003), automated scoring of spontaneous spoken language has proven to be more challenging and complex. Spoken lan- guage tests are still mostly scored by human rat- ers. However, several systems exist that score different aspects of spoken language; (Bernstein, 1999; C. Cucchiarini, H. Strik, & L. Boves, 1997a; Franco et al., 2000). Our work departs from previous research in that our goal is to study the feasibility of automating scoring for spontaneous speech, that is, when the spoken text is not known in advance. We approach scoring here as the characteri- zation of a speaker’s oral proficiency based on features that can be extracted from a spoken re- sponse to a well defined test question by means of automatic speech recognition (ASR). We further approach scoring as the construction of a mapping from a set of features to a score scale, in our case five discrete scores from 1 (least pro- ficient) to 5 (most proficient). The set of fea- tures and the specific mapping are motivated by the concept of communicative competence (Bachman, 1990; Canale & Swain, 1980; Hymes, 1972). This means that the features in the scoring system we are developing are meant to characterize specific components of commu- nicative competence, such as mastery of pronun- ciation, fluency, prosodic, lexical, grammatical and pragmatical subskills. The selection of fea- tures is guided by an understanding of the nature of speaking proficiency. We rely on the scoring behavior of judges to evaluate the features (sec- …

Content truncated to 500 words