A program for automatically selecting the best output from multiple machine translation engines

A Program for Automatically Selecting the Best Output from Multiple Machine Translation Engines Chris Callison-Burch, Raymond S. Flournoy Amikai, Inc. 343 Vermont Street San Francisco, CA 94103 USA ccb@amikai.com, rflournoy@amikai.com Abstract This paper describes a program that automatically selects the best translation from a set of translations produced by multiple commercial machine translation engines. The program is simplified by assuming that the most fluent item in the set is the best translation. Fluency is determined using a trigram language model. Results are provided illustrating how well the program performs for human ranked data as compared to each of its constituent engines. Keywords machine translation, language modeling, statistical methods, translation quality Introduction Amikai, Inc. develops machine translation applications. The technology is a combination of in-house and out- sourced material, with the bulk of the translation work coming from various third party translation engines. Amikai is a “best of breed” provider, which means that for each language pair that we support we provide access to the (single) best translation engine available on the market. This paper describes our efforts towards developing a system which uses multiple translation engines for each language pair, and dynamically chooses the best translation from a candidate set of translations for each input. We reasoned that if we built a program that could identify the best translation within a set then we would be able to claim that our quality was at least as good as the best translation engine used to produce the candidate set. Provided that one engine did not uniformly produce the best translation, then we would in fact have better overall quality than any given engine. The idea of building such a program begs the questions: Isn’t being able to automatically distinguish the quality of translations essentially as difficult as building a machine translation engine? In that case, why use third party translation engines at all? Building a fully-fledged translation engine is an enormous task. We’ve simplified the problem of choosing the best translation by making one crucial assumption: that the most fluent output corresponds to the best translation. With that assumption, the problem of choosing the best translation can be divorced from the input, and the problem is reduced to choosing the most fluent output. When translation engines fail to do a complete analysis of the input, their recovery strategies for producing a translation often result in “word salad”. If an engine produces a fluent translation then it is likely that the engine had a successful analysis of the input, increasing the likelihood that the meaning will be successfully transferred. We verified our assumption with human testing, and built a statistical language model to rank fluency automatically. Prior Work In order to automatically rank the translations that are produced by our collection of commercial translation engines, we assign a probability to each engine’s output with a statistical language model of the target language. Each language model judges the probability that each output is a sentence in that language. The highest ranking output is …

Content truncated to 500 words