A Flexible Online Server for Machine Translation Evaluation

A Flexible Online Server for Machine Translation Evaluation Matthias Eck, Stephan Vogel, and Alex Waibel InterACT Research Carnegie Mellon University Pittsburgh, PA, 15213, USA {matteck, vogel, waibel}@cs.cmu.edu Abstract. We present an Online Server for Machine Translation Evaluation that offers improvements over the standard usage of the typical scoring scripts. Users are able to interactively define their own test sets, experiments and pre-processing steps. Several scores are automatically calculated for submitted translations and the hypotheses and scores are organized and archived for later review. The server offers a nice web based user interface. 1. Introduction Evaluating machine translation hypotheses is a very important part of the ongoing research. Automatic scoring metrics allow a fast evaluation of translations and a quick turn- around for experiments. Researchers rely on the evaluation metrics to measure performance improvements gained by new approaches. The well-known automatic scores for machine translation are BLEU and NIST but a variety of other scores is available (Papineni, Roukos, Ward, and Zhu, 2002; Doddington, 2001; Banerjee and Lavie, 2005). Most of the scores rely on software programs or scripts that expect a variety of parameters including the hypothesis and reference files. The software then calculates the appropriate score. For most applications the files have to be in a special SGML file format that tags the different parts of the hypothesis or reference file. It is especially difficult for newcomers or for people who just want to get a glimpse of the possibilities to use these software programs. An experienced developer will most probably have a sophisticated setup for translation scoring but this will take a while for a beginner. The web server application presented here tries to circumvent some of the difficulties of scoring machine translation output. The online user interface offers an interactive environment in which test sets and experiments can be defined and hypotheses can be scored. The server stores the submitted translations for later review. It also offers directly accessible web services that allow score calculation in scripts and software programs based on the defined test sets. 2. Related Work Online Servers for Competitive Evaluations Different online servers have been used to evaluate translations for a variety of competitive evaluations. Especially notable are the evaluation servers for the NIST MT Evaluations (NIST, 2001-2006) and for the Evaluations in the International Workshops for Spoken Language Translation (IWSLT) in the years 2004 and 20051 (Akiba, Federico, Kando, Nakaiwa, Paul, and Tsuji, 2004; Eck and Hori, 2005). All of these evaluation servers were geared towards the needs of a competitive evaluation. The main goal was to make it easier for the organizers to handle a large amount of translation submissions and not necessarily to support the research of the participants. The servers did for example not show any scores during the actual evaluation period so that tuning the systems was impossible. The servers also did not provide any possibility to the participants to set up their own test sets. 1 The server presented here was developed based on the server used …

Content truncated to 500 words