Investigating the effect of auxiliary objectives for the automated grading of learner English speech transcriptions

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2258–2269 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2258 Investigating the Effect of Auxiliary Objectives for the Automated Grading of Learner English Speech Transcriptions Hannah Craighead1∗ Andrew Caines2 Paula Buttery2 Helen Yannakoudakis3 1 Computer Laboratory, University of Cambridge, U.K. hjc68@cl.cam.ac.uk 2 ALTA Institute & Computer Laboratory, University of Cambridge, U.K. {andrew.caines|paula.buttery}@cl.cam.ac.uk 3 Dept. of Informatics, King’s College London, U.K. helen.yannakoudakis@kcl.ac.uk Abstract We address the task of automatically grad- ing the language proficiency of spontaneous speech based on textual features from auto- matic speech recognition transcripts. Moti- vated by recent advances in multi-task learn- ing, we develop neural networks trained in a multi-task fashion that learn to predict the pro- ficiency level of non-native English speakers by taking advantage of inductive transfer be- tween the main task (grading) and auxiliary prediction tasks: morpho-syntactic labeling, language modeling, and native language iden- tification (L1). We encode the transcriptions with both bi-directional recurrent neural net- works and with bi-directional representations from transformers, compare against a feature- rich baseline, and analyse performance at dif- ferent proficiency levels and with transcrip- tions of varying error rates. Our best perfor- mance comes from a transformer encoder with L1 prediction as an auxiliary task. We discuss areas for improvement and potential applica- tions for text-only speech scoring. 1 Introduction The growing demand for the ability to commu- nicate in English means that both academic and commercial efforts are increasing to provide au- tomated tutoring and assessment systems. These educational systems address the increasing need for online resources to help students learn and to map users to the validated proficiency scales which play a critical role in securing education and work opportunities (British Council, 2013). Language learning applications delivered through smart speakers such as Amazon Alexa and Google Home are a novel form of educational technology. These offer obvious benefits to users in terms of immediacy, interaction and ∗ Currently at Google U.K. convenience. However, it remains challenging for application providers to assess language content collected through these means. Audio recordings are not returned to the developers for privacy reasons: instead only text responses are returned, the output of automated speech recognition (ASR) systems. This sets a new task in educational applications: the automated proficiency assessment of speech based on transcriptions alone. In this paper we report on our efforts to grade learner English transcriptions obtained from ASR systems, comparing a feature-rich baseline with neural networks trained on multi-task objectives. To assess spontaneous speech, automated grad- ing systems tend to use a combination of fea- tures extracted from the audio recording and the transcription resulting from ASR. For instance, SpeechRaterTM by the Educational Testing Service uses text-based features based on frequency counts and lexical unigrams – among others, the number of word tokens per second, the length of interpausal units in words, the vocabulary size normalized by recording duration – and score predictions are made using linear regression (Zechner et al., 2007, 2009; …

Content truncated to 500 words