An Empirical Study on Computing Consensus Translations from Multiple Machine Translation Systems

Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pp. 986–995, Prague, June 2007. c⃝2007 Association for Computational Linguistics An Empirical Study on Computing Consensus Translations from Multiple Machine Translation Systems Wolfgang Macherey Google Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043, USA wmach@google.com Franz Josef Och Google Inc. 1600 Amphitheatre Parkway Mountain View, CA 94043, USA och@google.com Abstract This paper presents an empirical study on how different selections of input translation systems affect translation quality in system combination. We give empirical evidence that the systems to be combined should be of similar quality and need to be almost uncorrelated in order to be beneficial for sys- tem combination. Experimental results are presented for composite translations com- puted from large numbers of different re- search systems as well as a set of transla- tion systems derived from one of the best- ranked machine translation engines in the 2006 NIST machine translation evaluation. 1 Introduction Computing consensus translations from the outputs of multiple machine translation engines has become a powerful means to improve translation quality in many machine translation tasks. Analogous to the ROVER approach in automatic speech recognition (Fiscus, 1997), a composite translation is computed by voting on the translation outputs of multiple machine translation systems. Depending on how the translations are combined and how the voting scheme is implemented, the composite translation may differ from any of the original hypotheses. While elementary approaches simply select for each sentence one of the original translations, more so- phisticated methods allow for combining transla- tions on a word or a phrase level. Although system combination could be shown to result in substantial improvements in terms of translation quality (Matusov et al., 2006; Sim et al., 2007), not every possible ensemble of translation outputs has the potential to outperform the primary translation system. In fact, an adverse combina- tion of translation systems may even deteriorate translation quality. This holds to a greater extent, when the ensemble of translation outputs contains a significant number of translations produced by low performing but highly correlated systems. In this paper we present an empirical study on how different ensembles of translation outputs affect performance in system combination. In particular, we will address the following questions: • To what extent can translation quality benefit from combining systems developed by multiple research labs? Despite an increasing number of translation engines, most state-of-the-art systems in statis- tical machine translation are nowadays based on implementations of the same techniques. For instance, word alignment models are often trained using the GIZA++ toolkit (Och and Ney, 2003); error minimizing training criteria such as the Minimum Error Rate Training (Och, 2003) are employed in order to learn feature function weights for log-linear models; and translation candidates are produced using phrase-based decoders (Koehn et al., 2003) in combination with n-gram language models (Brants et al., 2007). All these methods are established as de facto standards and form an integral part of most statistical machine translation systems. …

Content truncated to 500 words