Exploring Content Features for Automated Speech Scoring

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 103–111, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Exploring Content Features for Automated Speech Scoring Shasha Xie, Keelan Evanini, Klaus Zechner Educational Testing Service (ETS) Princeton, NJ 08541, USA {sxie,kevanini,kzechner}@ets.org Abstract Most previous research on automated speech scoring has focused on restricted, predictable speech. For automated scoring of unrestricted spontaneous speech, speech proficiency has been evaluated primarily on aspects of pro- nunciation, fluency, vocabulary and language usage but not on aspects of content and topi- cality. In this paper, we explore features repre- senting the accuracy of the content of a spoken response. Content features are generated us- ing three similarity measures, including a lex- ical matching method (Vector Space Model) and two semantic similarity measures (Latent Semantic Analysis and Pointwise Mutual In- formation). All of the features exhibit moder- ately high correlations with human proficiency scores on human speech transcriptions. The correlations decrease somewhat due to recog- nition errors when evaluated on the output of an automatic speech recognition system; how- ever, the additional use of word confidence scores can achieve correlations at a similar level as for human transcriptions. 1 Introduction Automated assessment of a non-native speaker’s proficiency in a given language is an attractive ap- plication of automatic speech recognition (ASR) and natural language processing (NLP) technology; the technology can be used by language learners for individual practice and by assessment providers to reduce the cost of human scoring. While much research has been done about the scoring of re- stricted speech, such as reading aloud or repeating sentences verbatim (Cucchiarini et al., 1997; Bern- stein et al., 2000; Cucchiarini et al., 2000; Witt and Young, 2000; Franco et al., 2000; Bernstein et al., 2010b), much less has been done about the scor- ing of spontaneous speech. For automated scor- ing of unrestricted, spontaneous speech, most auto- mated systems have estimated the non-native speak- ers’ speaking proficiency primarily based on low- level speaking-related features, such as pronuncia- tion, intonation, rhythm, rate of speech, and fluency (Cucchiarini et al., 2002; Zechner et al., 2007; Chen et al., 2009; Chen and Zechner, 2011a), although a few recent studies have explored features based on vocabulary and grammatical complexity (Zechner et al., 2007; Bernstein et al., 2010a; Bernstein et al., 2010b; Chen and Zechner, 2011b). To date, little work has been conducted on au- tomatically assessing the relatively higher-level as- pects of spontaneous speech, such as the content and topicality, the structure, and the discourse in- formation. Automated assessment of these aspects of a non-native speaker’s speech is very challeng- ing for a number of reasons, such as the short length of typical responses (approximately 100 words for a typical 1 minute response, compared to over 300 words in a typical essay/written response), the spon- taneous nature of the speech, and the presence of disfluencies and possible grammatical errors. More- over, the assessment system needs text transcripts of the speech to …

Content truncated to 500 words