Multitask Learning for Adaptive Quality Estimation of Automatically Transcribed Utterances

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 714–724, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics Multitask Learning for Adaptive Quality Estimation of Automatically Transcribed Utterances Jos´e G. C. de Souza⋆†, Hamed Zamani‡, Matteo Negri†, Marco Turchi†, Daniele Falavigna† ⋆University of Trento, Italy †Fondazione Bruno Kessler, Italy ‡School of ECE, College of Engineering, University of Tehran, Iran {desouza, negri, turchi, falavi}@fbk.eu h.zamani@ut.ac.ir Abstract We investigate the problem of predicting the quality of automatic speech recognition (ASR) output under the following rigid con- straints: i) reference transcriptions are not available, ii) confidence information about the system that produced the transcriptions is not accessible, and iii) training and test data come from multiple domains. To cope with these constraints (typical of the constantly increas- ing amount of automatic transcriptions that can be found on the Web), we propose a domain-adaptive approach based on multitask learning. Different algorithms and strategies are evaluated with English data coming from four domains, showing that the proposed ap- proach can cope with the limitations of previ- ously proposed single task learning methods. 1 Introduction The variety of applications for large vocabulary speech recognition technology (LVCSR) is rapidly growing. For instance, automatic transcriptions are now used, either as-is or as rough material to be checked and corrected by humans, for captioning and subtitling DVD movies, Youtube videos, TV programs and recordings in noisy environments such as meetings and teleconferences. To enable fur- ther integration in these and other scenarios, the im- provement of the core automatic speech recognition (ASR) technology should go hand in hand with the development of evaluation methods adequate to ad- dress new needs and constraints. Indeed, the stan- dard evaluation protocol, based on computing the word error rate of transcription hypotheses against reference transcripts,1 is not always a viable solu- tion. In terms of needs, the aforementioned appli- cations call for efficient and replicable evaluation methods suitable for real-time processing. While the availability of manually-created reference tran- scripts is a core ingredient for system development, tuning and lab testing, their use for on-field evalu- ation (i.e. during the actual use) is impractical for obvious reasons (i.e. the need of a quick response). In terms of constraints, the problem is that ASR technology is often used as a black-box, that is, with- out any knowledge of how the transcriptions are gen- erated.2 This calls for techniques capable to esti- mate ASR output quality under the rigid constraint of having, as a basic source of information, only the spoken utterance (the acoustic signal) and the tran- scription itself. Indeed, the invaluable information provided by current confidence estimation methods (e.g. word posterior probabilities (Evermann and Woodland, 2000; Wessel et al., 2001), consensus de- coding (Mangu et al., 2000) and minimum Bayes- risk decoding (Xu et al., 2010)) is not accessible when evaluating the output of an unknown system. 1The word error rate (WER) is the minimum edit distance between an hypothesis and the …

Content truncated to 500 words