Using the TED Talks to Evaluate Spoken Post-editing of Machine Translation

Using the TED Talks to Evaluate Spoken Post-editing of Machine Translation Jeevanthi Liyanapathirana Andrei Popescu-Belis Copenhagen Business School Idiap Research Institute Dalgas Have 15 Rue Marconi 19 DK-2000 Frederiksberg, Denmark CH-1920 Martigny, Switzerland jl.ibc@cbs.dk andrei.popescu-belis@idiap.ch Abstract This paper presents a solution to evaluate spoken post-editing of imperfect machine translation output by a human translator. We compare two approaches to the combination of machine translation (MT) and automatic speech recognition (ASR): a heuristic algorithm and a machine learning method. To obtain a data set with spoken post-editing information, we use the French version of TED talks as the source texts submitted to MT, and the spoken English counterparts as their corrections, which are submitted to an ASR system. We ex- periment with various levels of artificial ASR noise and also with a state-of-the-art ASR system. The results show that the combination of MT with ASR improves over both individual outputs of MT and ASR in terms of BLEU scores, especially when ASR performance is low. Keywords: machine translation, spoken post-editing, evaluation 1. Introduction In this paper, we consider the task of voice-based post- editing of the output of a machine translation (MT) sys- tem. In other words, a human corrects the translation pro- duced by a system simply by speaking out this correction. The paper presents a method for using an existing dataset to evaluate spoken post-editing methods without the need for costly experiments involving human post-editors. We use the French transcripts of TED talks (www.ted.com) as the source texts to be translated by MT, the original En- glish audio recordings as the spoken corrections, and the English transcripts as a reference for the automatic evalua- tion of translation quality. The proposed evaluation framework is applied to two orig- inal methods for combining the outputs of an automatic speech recognition (ASR) system and of an MT system, to generate a better translation than each of the outputs consid- ered individually. The first method is heuristic-based and relies on the confidence of the ASR to select words, while the second method uses machine learning to learn selec- tion rules based on a wider set of features, including word length, position and part-of-speech, and ASR confidence. The paper is organized as follows. Section 2 presents the motivation of our system. Section 3 describes the setting of our experiments, the ASR and MT systems, and the use of TED data for spoken post-editing. Section 4 defines the two approaches to ASR/MT combination that we use to gener- ate an improved translation. Section 5 describes the experi- ments and their results. Finally, Section 6 reviews previous work that has been done in this area, in comparison to our proposal. 2. Motivation Voice-based post-editing is valuable whenever using a key- board is not possible or practical, e.g. with mobile devices or impaired users. Therefore, the range of applications of our proposal is quite large, and covers the dissemination uses of MT rather than the assimilation ones, as defined by Hovy et al. (2002), because it helps …

Content truncated to 500 words