Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4506–4520 Barcelona, Spain (Online), December 8-13, 2020 4506 Is MAP Decoding All You Need? The Inadequacy of the Mode in Neural Machine Translation Bryan Eikema University of Amsterdam b.eikema@uva.nl Wilker Aziz University of Amsterdam w.aziz@uva.nl Abstract Recent studies have revealed a number of pathologies of neural machine translation (NMT) sys- tems. Hypotheses explaining these mostly suggest there is something fundamentally wrong with NMT as a model or its training algorithm, maximum likelihood estimation (MLE). Most of this evidence was gathered using maximum a posteriori (MAP) decoding, a decision rule aimed at identifying the highest-scoring translation, i.e. the mode. We argue that the evidence corroborates the inadequacy of MAP decoding more than casts doubt on the model and its training algorithm. In this work, we show that translation distributions do reproduce various statistics of the data well, but that beam search strays from such statistics. We show that some of the known patholo- gies and biases of NMT are due to MAP decoding and not to NMT’s statistical assumptions nor MLE. In particular, we show that the most likely translations under the model accumulate so little probability mass that the mode can be considered essentially arbitrary. We therefore advocate for the use of decision rules that take into account the translation distribution holistically. We show that an approximation to minimum Bayes risk decoding gives competitive results confirming that NMT models do capture important aspects of translation well in expectation. 1 Introduction Recent findings in neural machine translation (NMT) suggest that modern translation systems have some serious flaws. This is based on observations such as: i) translations produced via beam search typically under-estimate sequence length (Sountsov and Sarawagi, 2016; Koehn and Knowles, 2017), the length bias; ii) translation quality generally deteriorates with better approximate search (Koehn and Knowles, 2017; Murray and Chiang, 2018; Ott et al., 2018; Kumar and Sarawagi, 2019), the beam search curse; iii) the true most likely translation under the model (i.e., the mode of the distribution) is empty in many cases (Stahlberg and Byrne, 2019) and a general negative correlation exists between likelihood and qual- ity beyond a certain likelihood value (Ott et al., 2018), we call this the inadequacy of the mode problem. A number of hypotheses have been formulated to explain these observations. They mostly suggest there is something fundamentally wrong with NMT as a model (i.e., its factorisation as a product of locally normalised distributions) or its most popular training algorithm (i.e., regularised maximum likeli- hood estimation, MLE for short). These explanations make an unspoken assumption, namely, that iden- tifying the mode of the distribution, also referred to as maximum a posteriori (MAP) decoding (Smith, 2011), is in some sense the obvious decision rule for predictions. While this assumption makes intuitive sense and works well in unstructured classification problems, it is less justified in NMT, where often- times the most likely translations together account for very little probability mass, a claim we shall defend conceptually …

Content truncated to 500 words