Automatic Spontaneous Speech Grading: A Novel Feature Derivation Technique using the Crowd

Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1085–1094, Beijing, China, July 26-31, 2015. c⃝2015 Association for Computational Linguistics Automatic Spontaneous Speech Grading: A Novel Feature Derivation Technique using the Crowd Vinay Shashidhar Aspiring Minds vinay.shashidhar@aspiringminds.com Nishant Pandey Aspiring Minds nishant.pandey@aspiringminds.com Varun Aggarwal Aspiring Minds varun@aspiringminds.com Abstract In this paper, we address the problem of evaluating spontaneous speech us- ing a combination of machine learning and crowdsourcing. Machine learning techniques inadequately solve the stated problem because automatic speaker- independent speech transcription inaccurate. The features derived from itis are also inaccurate and so is the machine learning model developed for speech evaluation. To address this, we post the task of speech transcription to a large community of online workers (crowd). We also get spoken English grades from the crowd. We achieve 95% transcription accuracy by combining transcriptions from multiple crowd workers. Speech and prosody features are derived by force aligning the speech samples on these highly accurate transcriptions. Addi- tionally, we derive surface and semantic level features directly from the transcrip- tion. To demonstrate the efficacy of our approach we performed experiments on an expert–graded speech sample of 319 adult non–native speakers. Using these features in a regression model, we are able achieve a Pearson correlation of 0.76 with expert grades, an accuracy much higher than any previously reported machine learning approach. Our approach has an accuracy that rivals that of expert agreement. This work is timely given the huge requirement of spoken English training and assessment. 1 Introduction Automatic evaluation of spoken English has been of keen interest for more than two decades (Zech- ner et al., 2007; Neumeyer et al., 1996; Franco et al., 2000; Cucchiarini et al., 1997). It can help learners get feedback in a scalable manner, help build better English training software and also help companies and institutions filter and se- lect prospective employees more effectively. The problem acquires significance given the evidence that better English leads to better employment out- come, wages and promotions (Guven and Islam, 2013). There has been a considerable success in auto- matically scoring spoken English, when the spo- ken text is known a priori (Cucchiarini et al., 2000; Franco et al., 2000). In these cases, the candidate is asked to either read a given text or listen to some speech and repeat it. For these tasks, the scores generated by an automatic sys- tem on parameters such as pronunciation and flu- ency closely mimic those given by human ex- perts. The primary approach behind a majority of these systems is to force align the speech sample on the known text using an HMM–based acous- tic model. Features such as likelihood, posterior probability and fluency related features are derived from the aligned speech and a machine learning model is used to predict expert grades (Neumeyer et al., 1996; Franco et al., 2000; Cucchiarini et al., 1997). Some approaches additionally use prosody and energy related features (Dong …

Content truncated to 500 words