End-to-End Evaluation in Simultaneous Translation

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 345–353, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics End-to-End Evaluation in Simultaneous Translation Olivier Hamon1,2, Christian Fügen3, Djamel Mostefa1, Victoria Arranz1, Muntsin Kolss3, Alex Waibel3,4 and Khalid Choukri1 1Evaluations and Language Resources Distribution Agency (ELDA), Paris, France 2 LIPN (UMR 7030) – Université Paris 13 & CNRS, Villetaneuse, France 3 Univerität Karlsruhe (TH), Germany 4 Carnegie Mellon University, Pittsburgh, USA {hamon|mostefa|arranz|choukri}@elda.org, {fuegen|kolss|waibel}@ira.uka.de Abstract This paper presents the end-to-end evalu- ation of an automatic simultaneous trans- lation system, built with state-of-the-art components. It shows whether, and for which situations, such a system might be advantageous when compared to a human interpreter. Using speeches in English translated into Spanish, we present the evaluation procedure and we discuss the results both for the recognition and trans- lation components as well as for the over- all system. Even if the translation process remains the Achilles’ heel of the system, the results show that the system can keep at least half of the information, becoming potentially useful for final users. 1 Introduction Anyone speaking at least two different languages knows that translation and especially simultaneous interpretation are very challenging tasks. A human translator has to cope with the special nature of different languages, comprising phenomena like terminology, compound words, idioms, dialect terms or neologisms, unexplained acronyms or ab- breviations, proper names, as well as stylistic and punctuation differences. Further, translation or in- terpretation are not a word-by-word rendition of what was said or written in a source language. In- stead, the meaning and intention of a given sen- tence have to be reexpressed in a natural and fluent way in another language. Most professional full-time conference inter- preters work for international organizations like the United Nations, the European Union, or the African Union, whereas the world’s largest em- ployer of translators and interpreters is currently the European Commission. In 2006, the European Parliament spent about 300 million Euros, 30% of its budget, on the interpretation and translation of the parliament speeches and EU documents. Gen- erally, about 1.1 billion Euros are spent per year on the translating and interpreting services within the European Union, which is around 1% of the total EU-Budget (Volker Steinbiss, 2006). This paper presents the end-to-end evaluation of an automatic simultaneous translation system, built with state-of-the-art components. It shows whether, and in which cases, such a system might be advantageous compared to human interpreters. 2 Challenges in Human Interpretation According to Al-Khanji et al. (2000), researchers in the field of psychology, linguistics and interpre- tation seem to agree that simultaneous interpre- tation (SI) is a highly demanding cognitive task involving a basic psycholinguistic process. This process requires the interpreter to monitor, store and retrieve the input of the source language in a continuous manner in order to produce the oral rendition of this input in the target language. It is clear that this type of difficult linguistic and cog- nitive operation will …

Content truncated to 500 words