Evaluating Machine Translation Output with Automatic Sentence Segmentation

Overview of the IWSLT 2005 Evaluation Campaign Matthias Eck and Chiori Hori Interactive Systems Laboratories Carnegie Mellon University Pittsburgh, PA, 15213, USA {matteck, chiori}@cs.cmu.edu Abstract This paper reports an overview of the evaluation campaign results of the IWSLT 2005 workshop1. The BTEC corpus, which consists of typical travel domain phrases, was used. Data for the five language pairs Arabic/Chinese/Japanese/Korean to English and English to Chinese was prepared. To study how much the amount of the training data and how much different training and decoding approaches contribute to the performance, a supplied data and an unrestricted data track were introduced. In addition, translation results were evaluated not only for text input but also speech recognition output. 19 systems from 17 organizations participated in the evaluation. All machine translation results were evaluated using automatic evaluation metrics. The most popular track, translating text form Chinese to English, was graded by 3 humans in terms of Fluency, Adequacy and Meaning Maintenance. The correlation between automatic evaluation metrics and human judgment was examined. 1. Introduction The Consortium for Speech Translation Advanced Research (C-STAR) had been formed in the 1990s to study and develop techniques for speech-to-speech translation. To further this research C-STAR members have been jointly constructing a multilingual spoken language corpus, the basic travel expression corpus (BTEC, [1]). In 2004 the International Workshop on Spoken Language Translation (IWSLT) was started in order to enable the exchange of knowledge among researchers working on speech-to-speech translations and to create an opportunity to enhance the machine translation (MT) systems by comparing technologies on the same test bed [2]. IWSLT 2005 extended over the 2004 evaluation campaign by translating the output of automatic speech recognition (ASR) systems as well. Speech-to-speech translation systems are designed as systems combining an MT system with automatic speech recognition technology. This introduces additional difficulties into the translation process, 1 Workshop website: http://www.is.cs.cmu.edu/iwslt2005/ caused not only by disfluencies due to the spontaneity of the spoken language but also due to the errors in the ASR output. To accomplish speech-to-speech translation, we have to solve problems in translating spoken language and handling ASR output. This was the reason why we focused on handling speech recognition output including multiple recognition hypotheses this year. Translating speech with the goal of maintaining the original information in the source speech makes it necessary to handle recognition errors. Some of the problems that will have • to be addressed are: How can ASR output be translated more accurately even if recognition errors exist? • How much could the MT performance be enhanced by considering multiple hypotheses? • Which hypothesis contributes the most to the MT performance? • How to select the best hypothesis that can be translated well from multiple hypotheses? To alleviate the difficulty to work on both speech recognition and machine translation, we provided speech recognition results in this workshop. Future evaluations might also include a speech recognition part evaluating the attendees’ ASR systems. More realistic and difficult data such as spontaneous conversational speech could also be used. …

Content truncated to 500 words