Assessing the impact of speech recognition errors on machine translation quality

Assessing the Impact of Speech Recognition Errors on Machine Translation Quality Nicholas Ruiz nicruiz@fbk.eu Marcello Federico federico@fbk.eu Fondazione Bruno Kessler, Trento, 38122, Italy Abstract In spoken language translation, it is crucial that an automatic speech recognition (ASR) system produces outputs that can be adequately translated by a statistical machine translation (SMT) system. While word error rate (WER) is the standard metric of ASR quality, the assumption that each ASR error type is weighted equally is violated in a SMT system that relies on struc- tured input. In this paper, we outline a statistical framework for analyzing the impact of specific ASR error types on translation quality in a speech translation pipeline. Our approach is based on linear mixed-effects models, which allow the analysis of ASR errors on a translation qual- ity metric. The mixed-effects models take into account the variability of ASR systems and the difficulty of each speech utterance being translated in a specific experimental setting. We use mixed-effects models to verify that the ASR errors that compose the WER metric do not contribute equally to translation quality and that interactions exist between ASR errors that cu- mulatively affect a SMT system’s ability to translate an utterance. Our experiments are carried out on the English to French language pair using eight ASR systems and seven post-edited ma- chine translation references from the IWSLT 2013 evaluation campaign. We report significant findings that demonstrate differences in the contributions of specific ASR error types toward speech translation quality and suggest further error types that may contribute to translation difficulty. 1 Introduction Spoken language translation (SLT) systems are composed with an automatic speech recogni- tion (ASR) system that transcribes source language utterances and a machine translation (MT) system that translates the transcripts into a target language. While there is growing interest in constructing tightly-coupled ASR and MT systems that leverage joint training and optimization (He and Deng, 2012, 2013), the dominant approach is to construct a pipeline consisting of a MT system that decodes one or more ASR hypotheses (Ney, 1999; Matusov et al., 2006; Bertoldi et al., 2007; Casacuberta et al., 2008). The individual SLT components are trained and evaluated independently against local optimization metrics that fit each model to its local task, but they do not generalize to overall SLT quality. In particular, the de-facto automatic evaluation metric for speech recognition is Word Error Rate (WER), which classifies ASR errors into three categories corresponding to Levenshtein distance alignments (i.e. insertions, substitutions, and deletions) between a hypothesis and its reference. The linguistic features of the erroneous words and their relative positions in an utterance are not taken into account. In this paper, we investigate the impact of ASR errors on speech translation quality. In par- ticular, does each type of ASR error contribute equally to the performance degradation of MT Al-Onaizan & Simard (Eds.) Proceedings of AMTA 2014, vol. 1: MT Researchers Vancouver, BC © The Authors 261 outputs, or do specific classes of ASR errors more greatly inhibit the capacity of …

Content truncated to 500 words