Disentangling ASR and MT Errors in Speech Translation

Disentangling ASR and MT Errors in Speech Translation Ngoc-Tien Le ngoc-tien.le@univ-grenoble-alpes.fr Benjamin Lecouteux benjamin.lecouteux@univ-grenoble-alpes.fr Laurent Besacier laurent.besacier@univ-grenoble-alpes.fr University Grenoble Alpes, CNRS, Grenoble INP, LIG, F-38000 Grenoble, France Abstract The main aim of this paper is to investigate automatic quality assessment for spoken language translation (SLT). More precisely, we investigate SLT errors that can be due to transcription (ASR) or to translation (MT) modules. This paper investigates automatic detection of SLT er- rors using a single classifier based on joint ASR and MT features. We evaluate both 2-class (good/bad) and 3-class (good/badASR/badMT ) labeling tasks. The 3-class problem necessi- tates to disentangle ASR and MT errors in the speech translation output and we propose two label extraction methods for this non trivial step. This enables - as a by-product - qualitative analysis on the SLT errors and their origin (are they due to transcription or to translation step?) on our large in-house corpus for French-to-English speech translation. Index Terms: Spoken Language Translation, Automatic Speech Recognition, Confidence Esti- mation, Quality Estimation, ASR and MT errors detection. 1 Introduction This paper addresses a relatively new quality assessment task: error detection in spoken language translation (SLT) using both automatic speech recognition (ASR) features and ma- chine translation (MT) features. To our knowledge, the first attempts to design error detection for speech translation, using both ASR and MT features, are our own work (Besacier et al., 2014, 2015) which is further extended in this paper submission. Contributions (1) This paper extends previous work (Besacier et al., 2014, 2015) in 2- class (good/bad) error detection in SLT using a single classifier based on joint ASR and MT features (2) in order to disentangle ASR and MT errors in SLT, we extend error detection to a 3-class problem (good/badASR/badMT ) where we try to find the source of the SLT errors (3) two methods are compared for setting such 3-class labels on our corpus and a first attempt to automatically detect errors and their origin in a SLT output is presented at the end of this paper. Outline The outline of this paper goes simply as follows: Section 2 formalizes error detec- tion in SLT and presents our experimental setup. Section 3 proposes two methods to disentangle ASR and MT errors in SLT output and presents statistics on a large French-English corpus. Sec- tion 4 presents our 2-class and 3-class error detection results while section 5 concludes this work and gives some perspectives. Proceedings of MT Summit XVI, vol.1: Research Track Nagoya, Sep. 18-22, 2017 | p. 312 2 Automatic Error Detection in Speech Translation 2.1 Formalization A quality estimation (or error detection) component in speech translation solves the equa- tion: ˆq = argmax q {pSLT (q|xf, f, ˆe)} (1) where xf is the given signal in the source language; ˆe 1 = (e1, e2, ..., eN) is the most probable target language sequence from the spoken language translation (SLT) process; f = (f1, f2, ..., fM) is the transcription of xf; q = (q1, q2, ..., …

Content truncated to 500 words