Using word posterior probabilities in lattice translation

Overview of the IWSLT 2007 Evaluation Campaign Cameron S. Fordyce Center for the Evaluation of Language and Communication Technologies, Trento fordyce@celct.it Abstract In this paper we give an overview of the 2007 evaluation campaign for the International Workshop on Spoken Language Translation (IWSLT)1. As with previous evaluation campaigns, the primary focus of the workshop was the translation of spoken language in the travel domain. This year there were four language pairs; the translation of Chinese, Italian, Arabic, and Japanese into English. The input data consisted of the output of ASR systems for read speech and clean text. The exceptions were the challenge task of the Italian English language pair which used spontaneous speech ASR outputs and transcriptions and the Chinese English task which used only clean text. A new characteristic of this year’s evaluation campaign was an increased focus on the sharing of resources. Participants were requested to submit the data and supplementary resources used in building their systems so that the other participants might be able to take advantage of the same resources. A second new characteristic this year was the focus on the human evaluation of systems. Each primary run was judged in the human evaluation for every task using a straightforward ranking of systems. This year's workshop saw an increased participation over last year's workshop. This year 24 groups submitted runs to one or more of the tasks, compared to the 19 groups that submitted runs last year [1]. Automatic and human evaluation were carried out to measure MT performance under each condition, ASR system outputs for read speech, spontaneous travel dialogues, and clean text. 1. Introduction IWSLT is an MT evaluation campaign organized by the Consortium for Speech Translation Advanced Research (C- Star)2. This consortium provides a common framework to compare and improve the state-of-the-art speech-to-speech translation (SST) technologies[1]. C-Star has organized annual workshops with progressively more challenging SST tasks with Japanese, Chinese, Arabic, Italian into English. The 2004 IWSLT workshop focused on evaluation metrics for SST[2]. The 2005 IWSLT focused on the translation of ASR outputs from read-speech inputs[3]. The 2006 IWSLT workshop focused on spontaneous translation of Chinese into English, and the translation of read Japanese, Arabic, and Italian into English[1]. The theme of this year’s evaluation campaign remained the same as last year’s, the translation of spontaneous-speech input. As with last year, the evaluation tasks were divided into 1 http://iwslt07.itc.it 2 http://www.c-star.org/ two major groups, two “Challenge” tasks for spontaneous speech and two “Classical” tasks focusing on read-speech. The challenge tasks included the languages Chinese and Italian to English. The Chinese challenge task was structured to mirror last year’s CE challenge task. Unfortunately, due to the unavailability of new CE test data at the last moment, clean text was substituted. The Italian to English challenge task marked a departure from the previous year in that the spontaneous speech came from a collection of transcribed dialogues from travel agent and client interactions via telephone. The classical tasks included read speech for both Japanese to …

Content truncated to 500 words