On NMT Search Errors and Model Errors: Cat Got Your Tongue?

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3356–3362, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3356 On NMT Search Errors and Model Errors: Cat Got Your Tongue? Felix Stahlberg∗ and Bill Byrne University of Cambridge Department of Engineering Trumpington St, Cambridge CB2 1PZ, UK {fs439,wjb31}@cam.ac.uk Abstract We report on search errors and model errors in neural machine translation (NMT). We present an exact inference procedure for neural se- quence models based on a combination of beam search and depth-first search. We use our exact search to find the global best model scores under a Transformer base model for the entire WMT15 English-German test set. Sur- prisingly, beam search fails to find these global best model scores in most cases, even with a very large beam size of 100. For more than 50% of the sentences, the model in fact assigns its global best score to the empty translation, revealing a massive failure of neural models in properly accounting for adequacy. We show by constraining search with a minimum trans- lation length that at the root of the problem of empty translations lies an inherent bias to- wards shorter translations. We conclude that vanilla NMT in its current form requires just the right amount of beam search errors, which, from a modelling perspective, is a highly un- satisfactory conclusion indeed, as the model often prefers an empty translation. 1 Introduction Neural machine translation (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Bahdanau et al., 2015, NMT) assigns the probability P(y|x) of a translation y = yJ 1 ∈ T J of length J over the target language vocabulary T for a source sen- tence x ∈ SI of length I over the source language vocabulary S via a left-to-right factorization using the chain rule: log P(y|x) = J � j=1 log P(yj|yj−1 1 , x). (1) The task of finding the most likely translation ˆy ∈ T ∗ for a given source sentence x is known as the *Now at Google. decoding or inference problem: ˆy = arg max y∈T ∗ P(y|x). (2) The NMT search space is vast as it grows expo- nentially with the sequence length. For example, for a common vocabulary size of |T | = 32, 000, there are already more possible translations with 20 words or less than atoms in the observable universe (32, 00020 ≫ 1082). Thus, complete enumeration of the search space is impossible. The size of the NMT search space is perhaps the main reason why – besides some preliminary stud- ies (Niehues et al., 2017; Stahlberg et al., 2018b; Ott et al., 2018) – analyzing search errors in NMT has received only limited attention. To the best of our knowledge, none of the previous studies were able to quantify the number of search errors in un- constrained NMT due to the lack of an exact infer- ence scheme that – although too …

Content truncated to 500 words