High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics

High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics Markus Freitag, David Grangier, Qijun Tan, Bowen Liang Google Research, USA {freitag, grangier, qijuntan, bowenl}@google.com Abstract In Neural Machine Translation, it is typically assumed that the sentence with the high- est estimated probability should also be the translation with the highest quality as mea- sured by humans. In this work, we question this assumption and show that model esti- mates and translation quality only vaguely correlate. We apply Minimum Bayes Risk (MBR) decoding on unbiased samples to opti- mize diverse automated metrics of translation quality as an alternative inference strategy to beam search. Instead of targeting the hy- potheses with the highest model probability, MBR decoding extracts the hypotheses with the highest estimated quality. Our experiments show that the combination of a neural trans- lation model with a neural reference-based metric, BLEURT, results in significant improve- ment in human evaluations. This improvement is obtained with translations different from classical beam-search output: These transla- tions have much lower model likelihood and are less favored by surface metrics like BLEU. 1 Introduction Neural sequence-to-sequence models constitute the state-of-the-art for machine translation. These models estimate the probability of a target sen- tence given a source sentence. At inference, it is commonplace to approximate the maximum-a- posteriori (MAP) hypothesis with beam search in order to output a sentence with (close to) the highest probability given the provided source. This strategy assumes that the sentences with the highest estimated probabilities should also be the translations with the highest quality as measured by humans. This assumption can be questioned based on two observations: (i) Neu- ral Machine Translations (NMTs) generated by beam search are ranked below human translations in professional evaluations (Freitag et al., 2021a) while (ii) the NMT model itself considers human translations much less likely than its beam outputs (Ott et al., 2018). These observations clearly show that estimated probability and translation quality do not always correlate. An example is given in Table 1, where beam search generates a transla- tion using mostly frequent words which results in inaccuracies. The two correct human translations contain infrequent words and phrases with low estimated probabilities based on the model. These observations do not in themselves sug- gest an alternative to likelihood in selecting better hypotheses. For that, we look at recent progress in automated evaluation. Recently introduced utility metrics, such as BLEURT (Sellam et al., 2020a) or COMET (Rei et al., 2020), estimate human judg- ments u(h, r) from a candidate translation h and a reference human translation r with a neural net- work. These learned metrics have shown higher correlation with human judgments compared with traditional metrics based on lexical overlap such as BLEU (Papineni et al., 2002) and METEOR (Banerjee and Lavie, 2005). BLEURT and COMET have also been shown by the WMT metric task (Freitag et al., 2021b) to perform better than YiSi (Lo, 2020), which measures overlap in a neural embedding space. BLEURT and COMET are able to …

Content truncated to 500 words