Off-topic Response Detection for Spontaneous Spoken English Assessment

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1075–1084, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Off-topic Response Detection for Spontaneous Spoken English Assessment Andrey Malinin, Rogier C. Van Dalen, Yu Wang, Kate M. Knill, Mark J. F. Gales University of Cambridge, Department of Engineering Trumpington St, Cambridge CB2 1PZ, UK {am969, yw396, kate.knill, mjfg}@eng.cam.ac.uk Abstract Automatic spoken language assessment systems are becoming increasingly impor- tant to meet the demand for English sec- ond language learning. This is a challeng- ing task due to the high error rates of, even state-of-the-art, non-native speech recog- nition. Consequently current systems pri- marily assess fluency and pronunciation. However, content assessment is essential for full automation. As a first stage it is important to judge whether the speaker re- sponds on topic to test questions designed to elicit spontaneous speech. Standard ap- proaches to off-topic response detection assess similarity between the response and question based on bag-of-words represen- tations. An alternative framework based on Recurrent Neural Network Language Models (RNNLM) is proposed in this pa- per. The RNNLM is adapted to the topic of each test question. It learns to asso- ciate example responses to questions with points in a topic space constructed using these example responses. Classification is done by ranking the topic-conditional posterior probabilities of a response. The RNNLMs associate a broad range of re- sponses with each topic, incorporate se- quence information and scale better with additional training data, unlike standard methods. On experiments conducted on data from the Business Language Testing Service (BULATS) this approach outper- forms standard approaches. 1 Introduction As English has become the global lingua franca, there is growing demand worldwide for assess- ment of English as a second language (Seidlhofer, 2005). To assess spoken communication, sponta- neous speech is typically elicited through a series of questions such as ’describe the photo’ or ’plan a meeting’. Grades are awarded based on a candi- date’s responses. Automatic assessment systems are becoming at- tractive as they allow second language assessment programmes to economically scale their opera- tions while decreasing throughput time and pro- vide testing on demand. Features for automatic graders are derived from the audio and from hy- potheses produced by automatic speech recogni- tion (ASR) systems. The latter is highly errorful due to the large variability in the input speech; disfluencies common to spontaneous speech, non- native accents and pronunciations. Current sys- tems, such as ETS’ SpeechRater (Zechner et al., 2009) and Pearson’s AZELLA (Metallinou and Cheng, 2014), primarily assess pronunciation and fluency. Although these are clearly indicative of spoken language ability, full assessment of spo- ken communication requires judgement of high- level content and communication skills, such as re- sponse construction and relevance. The first stage of this is to assess whether the responses are off- topic, that is, has the candidate misunderstood the question and/or memorised a response. While there has been little work done on de- tecting off-topic responses for spoken language assessment, detection of off-topic …

Content truncated to 500 words