Digging Errors in NMT: Evaluating and Understanding Model Errors from Partial Hypothesis Space

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 12067 - 12085 December 7-11, 2022 ©2022 Association for Computational Linguistics Digging Errors in NMT: Evaluating and Understanding Model Errors from Partial Hypothesis Space Jianhao Yan∗ WeChat AI, Tencent, China elliottyan37@gmail.com Chenming Wu∗ Tencent, China wcm1994@gmail.com Fandong Meng WeChat AI, Tencent, China fandongmeng@tencent.com Jie Zhou WeChat AI, Tencent, China withtomzhou@tencent.com Abstract Solid evaluation of neural machine translation (NMT) is key to its understanding and improve- ment. Current evaluation of an NMT system is usually built upon a heuristic decoding algo- rithm (e.g., beam search) and an evaluation met- ric assessing similarity between the translation and golden reference. However, this system- level evaluation framework is limited by evalu- ating only one best hypothesis and search errors brought by heuristic decoding algorithms. To better understand NMT models, we propose a novel evaluation protocol, which defines model errors with model’s ranking capability over hy- pothesis space. To tackle the problem of expo- nentially large space, we propose two approx- imation methods, top region evaluation along with an exact top-k decoding algorithm, which finds top-ranked hypotheses in the whole hy- pothesis space, and Monte Carlo sampling eval- uation, which simulates hypothesis space from a broader perspective. To quantify errors, we define our NMT model errors by measuring distance between the hypothesis array ranked by the model and the ideally ranked hypothesis array. After confirming the strong correlation with human judgment, we apply our evalua- tion to various NMT benchmarks and model architectures. We show that the state-of-the-art Transformer models face serious ranking issues and only perform at the random chance level in the top region. We further analyze model er- rors on architectures with different depths and widths, as well as different data-augmentation techniques, showing how these factors affect model errors. Finally, we connect model errors with the search algorithms and provide interest- ing findings of beam search inductive bias and correlation with Minimum Bayes Risk (MBR) decoding. 1 ∗Equal contribution. 1Source code: https://github.com/ ElliottYan/digging_errors_nmt 1 Introduction Sequence-to-sequence models (Sutskever et al., 2014; Vaswani et al., 2017) have shown promising results in neural machine translation (NMT), where methods typically frame a conditional probability distribution from a source sentence to a target sen- tence. One key to the booming of neural machine translation is the sound evaluation, which shows the trajectory to a better model design and architec- ture. The commonly used evaluation protocol of an NMT system comprises two main components: a search algorithm and an evaluation metric. The algorithm is responsible for decoding a translated sentence, and the metric computes the discrepancy between the generated translation and the refer- ence. The above evaluation protocol is preferred as it is consistent with what we serve in production NMT. It has an underlying assumption that the gap between an NMT model and the ideal model can be depicted by the gap between decoded translations and references. However, this assumption does not always hold. Recent literature (Stahlberg and Byrne, 2019; Meister et al., …

Content truncated to 500 words