Joint ASR and MT Features for Quality Estimation in Spoken Language Translation

Joint ASR and MT Features for Quality Estimation in Spoken Language Translation Ngoc-Tien Le, Benjamin Lecouteux and Laurent Besacier GETALP – LIG, University of Grenoble Alpes, France firstname.lastname@imag.fr Abstract This paper aims to unravel the automatic quality assess- ment for spoken language translation (SLT). More precisely, we propose several effective estimators based on our estima- tion of transcription (ASR) quality, translation (MT) quality, or both (combined and joint features using ASR and MT in- formation). Our experiments provide an important opportu- nity to advance the understanding of the prediction quality of words in a SLT output that were revealed by MT and ASR features. These results could be applied to interactive speech translation or computer-assisted translation of speeches and lectures. For reproducible experiments, the code allowing to call our WCE-LIG application and the corpora used are made available to the research community. 1. Introduction Automatic quality assessment of spoken language trans- lation (SLT), also named confidence estimation (CE), is an important topic because it allows to know if a system pro- duces (or not) user-acceptable outputs. In interactive speech to speech translation, CE helps to judge if a translated turn is uncertain (and ask the speaker to rephrase or repeat). For speech-to-text applications, CE may tell us if output transla- tions are worth being corrected or if they require retranslation from scratch. Moreover, an accurate CE can also help to im- prove SLT itself through a second-pass N-best list re-ranking or search graph re-decoding, as it has already been done for text translation in [1] and [2], or for speech translation in [3]. Consequently, building a method which is capable of poin- ting out the correct parts as well as detecting the errors in a speech translated output is crucial to tackle above issues. Outline The outline of this paper goes simply as fol- lows: section 2 reviews the state-of-the-art on confidence es- timation for ASR and MT. Our word confidence estimation (WCE) system using multiple features is then described in section 3. The experimental setup (notably our specific WCE corpus) is presented in section 4 while section 5 evaluates our joint WCE system and finally, section 6 concludes this work and gives some perspectives. 2. Related Work on Confidence Estimation for ASR and MT Several previous works tried to propose effective confi- dence measures in order to detect errors on ASR outputs. Confidence measures are introduced for Out-Of-Vocabulary (OOV) detection by [4]. [5] extends the previous work and introduces the use of word posterior probability (WPP) as a confidence measure for speech recognition. Posterior proba- bility of a word is most of the time computed using the hypo- thesis word graph [6]. Also, more recent approaches [7] for confidence measure estimation use side-information extrac- ted from the recognizer: normalized likelihoods (WPP), the number of competitors at the end of a word (hypothesis den- sity), decoding process behavior, linguistic features, acoustic features (acoustic stability, duration features) and semantic features. In parallel, the Workshop on Machine Translation (WMT) introduced in 2013 …

Content truncated to 500 words