Improved pronunciation features for construct-driven assessment of non-native spontaneous speech

Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the ACL, pages 442–449, Boulder, Colorado, June 2009. c⃝2009 Association for Computational Linguistics Improved Pronunciation Features for Construct-driven Assessment of Non-native Spontaneous Speech Lei Chen, Klaus Zechner, Xiaoming Xi Educational Testing Service Princeton, NJ, USA {LChen,KZechner,XXi}@ets.org Abstract This paper describes research on automatic as- sessment of the pronunciation quality of spon- taneous non-native adult speech. Since the speaking content is not known prior to the assessment, a two-stage method is developed to first recognize the speaking content based on non-native speech acoustic properties and then forced-align the recognition results with a reference acoustic model reflecting native and near-native speech properties. Features related to Hidden Markov Model likelihoods and vowel durations are extracted. Words with low recognition confidence can be excluded in the extraction of likelihood-related fea- tures to minimize erroneous alignments due to speech recognition errors. Our experiments on the TOEFL R⃝Practice Online test, an En- glish language assessment, suggest that the recognition/forced-alignment method can pro- vide useful pronunciation features. Our new pronunciation features are more meaningful than an utterance-based normalized acoustic model score used in previous research from a construct point of view. 1 Introduction Automated systems for evaluating highly pre- dictable speech (e.g. read speech or speech that is quite constrained in the use of vocabulary and syntactic structures) have emerged in the past decade (Bernstein, 1999; Witt, 1999; Franco et al., 2000; Hacker et al., 2005) due to the growing matu- rity of speech recognition and processing technolo- gies. However, endeavors into automated scoring for spontaneous speech have been sparse given the challenge of both recognizing and assessing spon- taneous speech. This paper addresses the develop- ment and evaluation of pronunciation features for an automated system for scoring spontaneous speech. This system was deployed for the TOEFL R⃝Practice Online (TPO) assessment used by prospective test takers to prepare for the official TOEFL R⃝test. A construct is a set of knowledge, skills, and abil- ities measured by a test. The construct of the speak- ing test is embodied in the rubrics that human raters use to score the test. It consists of three key cat- egories: delivery, language use, and topic devel- opment. Delivery refers to the pace and the clar- ity of the speech, including performance on into- nation, rhythm, rate of speech, and degree of hesi- tancy. Language use refers to the range, complex- ity, and precision of vocabulary and grammar use. Topic development refers to the coherence and full- ness of the response. Most of the research on spon- taneous speech assessment focuses on the delivery aspect given the low recognition accuracy on non- native spontaneous speech. The delivery aspect can be measured on four di- mensions: fluency, intonation, rhythm, and pronun- ciation. For the TPO assessment, we have defined pronunciation as the quality of vowels, consonants and word-level stress (segmentals). Intonation and sentence-level stress patterns (supra-segmentals) are not defined as part of pronunciation. Pronuncia- tion is one of the key factors …

Content truncated to 500 words