Generalized Minimum Bayes Risk System Combination

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 1356–1360, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Generalized Minimum Bayes Risk System Combination Kevin Duh, Katsuhito Sudoh, Xianchao Wu, Hajime Tsukada, Masaaki Nagata NTT Communication Science Laboratories 2-4 Hikari-dai, Seika-cho, Kyoto 619-0237, JAPAN kevin.duh@lab.ntt.co.jp Abstract Minimum Bayes Risk (MBR) has been used as a decision rule for both single- system decoding and system combination in machine translation. For system com- bination, we argue that common MBR implementations are actually not correct, since probabilities in the hypothesis space cannot be reliably estimated. These imple- mentations achieve the effect of consensus decoding (which may be beneficial in its own right), but does not reduce Bayes Risk in the true Bayesian sense. We introduce Generalized MBR, which parameterizes the loss function in MBR and allows it to be optimized in the given hypothesis space of multiple sys- tems. This extension better approximates the true Bayes Risk decision rule and empirically improves over MBR, even in cases where the combined systems are of mixed quality. 1 Introduction Minimum Bayes Risk (MBR) is a theoretically- elegant decision rule that has been used for single- system decoding and system combination in ma- chine translation (MT). MBR arose in Bayes deci- sion theory (Duda et al., 2000) and has since been applied to speech recognition (Goel and Byrne, 2000) and machine translation (Kumar and Byrne, 2004). The idea is to choose hypotheses that mini- mize Bayes Risk as oppose to those that maximize posterior probability. This enables the use of task- specific loss functions (e.g BLEU). However, the definition of Bayes Risk depends critically on the posterior probability of hypothe- ses. In single-system decoding, one could approx- imate this probability using model scores. How- ever, for system combination, the various systems have incompatible scores. In practice, MT design- ers resort to uniform probability, but the result is that the chosen hypothesis no longer has anything to do with Bayes Risk. This hypothesis can be seen as a consensus of multiple hypotheses, and in practice the consensus translation is often good, but it cannot be accurately thought of as MBR. Here, we propose a method that better achieves MBR in system combination settings. The insight is to generalize the loss function in the MBR equa- tion and allow it to be parameterized. The param- eters are then tuned on a small development data so that the loss function is converted to one that gives low Bayes Risk under the assumption of uni- form posteriors. We will show that a small bitext is sufficient for tuning this generalized loss, and that it vastly outperforms the conventional MBR approach in system combination. In the following, we first review MBR and ex- plain the difficulty in applying it to system combi- nation (Section 2). Then, we propose our Gener- alized MBR (Section 3) and evaluate it under the NTCIR Patent Translation tasks (Section 4). Fi- nally, we conclude in Section 5. 2 The Difficulty with …

Content truncated to 500 words