A Unified Approach to Minimum Risk Training and Decoding

Proceedings of the Joint 5th Workshop on Statistical Machine Translation and MetricsMATR, pages 365–374, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics A Unified Approach to Minimum Risk Training and Decoding Abhishek Arun, Barry Haddow and Philipp Koehn School of Informatics University of Edinburgh Edinburgh, EH8 9AB, UK a.arun@sms.ed.ac.uk, {bhaddow,pkoehn}@inf.ed.ac.uk Abstract We present a unified approach to perform- ing minimum risk training and minimum Bayes risk (MBR) decoding with BLEU in a phrase-based model. Key to our ap- proach is the use of a Gibbs sampler that allows us to explore the entire probabil- ity distribution and maintain a strict prob- abilistic formulation across the pipeline. We also describe a new sampling algo- rithm called corpus sampling which al- lows us at training time to use BLEU in- stead of an approximation thereof. Our approach is theoretically sound and gives better (up to +0.6%BLEU) and more sta- ble results than the standard MERT opti- mization algorithm. By comparing our ap- proach to lattice MBR, we are also able to gain crucial insights about both methods. 1 Introduction According to statistical decision theory, the opti- mal decision rule for any statistical model is the solution that minimizes its risk (expected loss). This solution is often referred to as the Minimum Bayes Risk (MBR) solution (Kumar and Byrne, 2004). Since machine translation (MT) mod- els are typically evaluated by BLEU (Papineni et al., 2002), a loss function which rewards partial matches, the MBR solution is to be preferred to the Maximum A Posteriori (MAP) solution. In most statistical MT (SMT) systems, MBR is implemented as a reranker of a list1 of trans- lations generated by a first-pass decoder. This de- coder typically assigns unnormalised log probabil- ities (known as scores) to each translation hypoth- 1We use the term list to denote any enumerable represen- tation of translation hypotheses e.g n-best list, translation lat- tice or forest. esis, so these scores must be converted to proba- bilities in order to apply MBR. In order to perform this conversion, it is first necessary to compute the normalization function Z. Since Z is defined as an intractable sum over all possible translations, it is approximated by summing over the translations in the list. The second step is to find the correct scale factor for the scores using a hyper-parameter search over held-out data. This is needed because the model parameters for the first-pass decoder are normally learnt using MERT (Och, 2003), which is invariant under scaling of the scores. Both these steps are theoretically unsatisfactory methods of estimating the posterior probability distribution since the approximation to Z is an un- bounded term and the scaling factor is an artificial way of inducing a probability distribution. Recently, (Tromble et al., 2008; Kumar et al., 2009) have shown that using a search lattice to im- prove the estimation of the true probability distri- bution can lead to improved MBR performance. However, these approaches still rely on MERT for training the base model, and in fact …

Content truncated to 500 words