Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10978 - 10993 December 7-11, 2022 ©2022 Association for Computational Linguistics Sampling-Based Approximations to Minimum Bayes Risk Decoding for Neural Machine Translation Bryan Eikema University of Amsterdam b.eikema@uva.nl Wilker Aziz University of Amsterdam w.aziz@uva.nl Abstract In NMT we search for the mode of the model distribution to form predictions. The mode and other high-probability translations found by beam search have been shown to often be inadequate in a number of ways. This pre- vents improving translation quality through bet- ter search, as these idiosyncratic translations end up selected by the decoding algorithm, a problem known as the beam search curse. Re- cently, an approximation to minimum Bayes risk (MBR) decoding has been proposed as an alternative decision rule that would likely not suffer from the same problems. We analyse this approximation and establish that it has no equivalent to the beam search curse. We then design approximations that decouple the cost of exploration from the cost of robust estimation of expected utility. This allows for much larger hypothesis spaces, which we show to be benefi- cial. We also show that mode-seeking strategies can aid in constructing compact sets of promis- ing hypotheses and that MBR is effective in identifying good translations in them. We con- duct experiments on three language pairs vary- ing in amounts of resources available: English into and from German, Romanian, and Nepali.1 1 Introduction NMT systems (Sutskever et al., 2014; Bahdanau et al., 2015) are trained to predict a conditional probability distribution over translation candidates of any given source sentence. After training, choos- ing a translation for a given input requires a deci- sion rule: a criterion to elect a ‘preferred’ transla- tion. MAP decoding, the most common decision rule in NMT, seeks the most probable translation under the model (i.e., the mode of the distribution). 1Code is available at github.com/roxot/mbr-nmt. Figure 1: NMT spreads probability roughly uniformly over a large set of promising hypotheses (left). MBR (right) assigns hypotheses an expected utility, revealing clear preferences against those that are too idiosyncratic. MAP decoding and its approximations such as beam search (Graves, 2012) have been under scrutiny. Stahlberg and Byrne (2019) show that the true mode is oftentimes inadequately short or empty. Better approximate search is known to hurt quality (Koehn and Knowles, 2017; Murray and Chiang, 2018; Kumar and Sarawagi, 2019), a prob- lem known as the beam search curse. The suc- cess of beam search depends on search biases in- troduced by hyperparameters such as beam size and length normalisation, which are tuned not to correlate with the objective of MAP decoding, but rather to strike a compromise between mode- seeking search and properties of reasonable trans- lations. Despite its success, a number of problems have been observed: length bias (Cho et al., 2014; Sountsov and Sarawagi, 2016), word frequency bias (Ott et al., 2018), susceptibility to copy noise (Khayrallah and Koehn, 2018; Ott et al., 2018), and hallucination under domain …

Content truncated to 500 words