Quality-Aware Decoding for Neural Machine Translation

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1396 - 1412 July 10-15, 2022 ©2022 Association for Computational Linguistics Quality-Aware Decoding for Neural Machine Translation Patrick Fernandes∗,1,2,3 António Farinhas∗,2,3 Ricardo Rei2,4,5 José G. C. de Souza5 Perez Ogayo1 Graham Neubig1 André F. T. Martins2,3,5 1Carnegie Mellon University 2Instituto Superior Técnico (Lisbon ELLIS Unit) 3Instituto de Telecomunicações 4INESC-ID 5Unbabel pfernand@cs.cmu.edu antonio.farinhas@tecnico.ulisboa.pt Abstract Despite the progress in machine translation quality estimation and evaluation in the last years, decoding in neural machine translation (NMT) is mostly oblivious to this and cen- ters around finding the most probable trans- lation according to the model (MAP decoding), approximated with beam search. In this pa- per, we bring together these two lines of re- search and propose quality-aware decoding for NMT, by leveraging recent breakthroughs in reference-free and reference-based MT evalu- ation through various inference methods like N-best reranking and minimum Bayes risk de- coding. We perform an extensive comparison of various possible candidate generation and ranking methods across four datasets and two model classes and find that quality-aware de- coding consistently outperforms MAP-based decoding according both to state-of-the-art au- tomatic metrics (COMET and BLEURT) and to human assessments. Our code is available at https://github.com/deep-spin/ qaware-decode. 1 Introduction The most common procedure in neural machine translation (NMT) is to train models using maxi- mum likelihood estimation (MLE) at training time, and to decode with beam search at test time, as a way to approximate maximum-a-posteriori (MAP) decoding. However, several works have questioned the utility of model likelihood as a good proxy for translation quality (Koehn and Knowles, 2017; Ott et al., 2018; Stahlberg and Byrne, 2019; Eikema and Aziz, 2020). In parallel, significant progress has been made in methods for quality estimation and evaluation of generated translations (Specia et al., 2020; Mathur et al., 2020b), but this progress is, by and large, not yet reflected in either training or decoding methods. Exceptions such as minimum risk training (Shen et al., 2016; Edunov et al., 2018) * Equal contribution. Figure 1: Quality-aware decoding framework. First, translation candidates are generated according to the model. Then, using reference-free and/or reference- based MT metrics, these candidates are ranked, and the highest ranked one is picked as the final translation. come at a cost of more expensive and unstable train- ing, often with modest quality improvements. An appealing alternative is to modify the decod- ing procedure only, separating it into two stages: candidate generation (§2.1; where candidates are generated with beam search or sampled from the whole distribution) and ranking (§2.2; where they are scored using a quality metric of interest, and the translation with the highest score is picked). This strategy has been explored in approaches using N-best reranking (Ng et al., 2019; Bhattacharyya et al., 2021) and minimum Bayes risk (MBR) de- coding (Shu and Nakayama, 2017; Eikema and Aziz, 2021; Müller and Sennrich, 2021). While this previous work has exhibited promising results, it has mostly focused …

Content truncated to 500 words