Quality Estimation for Automatic Speech Recognition

Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1813–1823, Dublin, Ireland, August 23-29 2014. Quality Estimation for Automatic Speech Recognition Matteo Negri(1) Marco Turchi(1) Jos´e G. C. de Souza(1,2) Daniele Falavigna(1) (1) FBK - Fondazione Bruno Kessler, Via Sommarive 18, 38123 Trento, Italy (2) University of Trento, Italy {negri,turchi,desouza,falavi}@fbk.eu Abstract We address the problem of estimating the quality of Automatic Speech Recognition (ASR) out- put at utterance level, without recourse to manual reference transcriptions and when information about system’s confidence is not accessible. Given a source signal and its automatic transcription, we approach this problem as a regression task where the word error rate of the transcribed utter- ance has to be predicted. To this aim, we explore the contribution of different feature sets and the potential of different algorithms in testing conditions of increasing complexity. Results show that our automatic quality estimates closely approximate the word error rate scores calculated over reference transcripts, outperforming a strong baseline in all the testing conditions. 1 Introduction In recent years, the increasing usage of large vocabulary continuous speech recognition (LVCSR) systems to transcribe audio recordings from different sources (e.g. Youtube videos, TV programs, DVD movies, meetings, etc) has sparked the need of accurate, fast and cost-effective methods to estimate the quality of ASR output. This need contrasts with the fact that, after decades of progress in ASR research, the established evaluation protocol is based on computing word error rate scores (WER)1 over large test sets of hand-crafted reference transcriptions. Indeed, despite its reliability, reference-based performance assessment has an evident drawback represented by the cost of acquiring manual transcripts. Besides increasing the cost-effectiveness of ASR evaluation routines, bypassing this bottleneck has several other motivations. From an application perspective, for instance, reference-free quality estimation methods could be used to: i) decide at run-time whether a given input signal has been properly recognized (e.g. if a user spoken utterance needs to be repeated in a dialogue application), ii) decide if an automatic transcription is acceptable as is (e.g. if manual revision is needed in an automatic subtitling application), or iii) select the best transcription among options from multiple ASR systems. When information about the inner workings of the system used to produce the transcriptions is acces- sible, current reference-free confidence estimation methods can supply ASR applications with reliable indicators about output reliability. This condition, however, does not always hold in the aforementioned scenarios. A clear motivating example is provided by the exponential growth of captioned TED Talks and Youtube videos,2 for which no information is available about how transcriptions have been pro- duced. In this case, neither reference-based methods, nor standard confidence measures can be applied to obtain useful quality estimates. Nevertheless, in this scenario, supplying reliable indicators of tran- scription quality has a huge market potential (e.g. to reduce the costs of manual revision/translation) which motivates our research. Focusing on these compelling needs, this paper investigates the automatic prediction of ASR out- put quality when: i) manual reference …

Content truncated to 500 words