Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7236–7249 August 1–6, 2021. ©2021 Association for Computational Linguistics 7236 Beyond Noise: Mitigating the Impact of Fine-grained Semantic Divergences on Neural Machine Translation Eleftheria Briakou and Marine Carpuat Department of Computer Science University of Maryland College Park, MD 20742, USA ebriakou@cs.umd.edu, marine@cs.umd.edu Abstract While it has been shown that Neural Machine Translation (NMT) is highly sensitive to noisy parallel training samples, prior work treats all types of mismatches between source and tar- get as noise. As a result, it remains unclear how samples that are mostly equivalent but contain a small number of semantically diver- gent tokens impact NMT training. To close this gap, we analyze the impact of different types of fine-grained semantic divergences on Trans- former models. We show that models trained on synthetic divergences output degenerated text more frequently and are less confident in their predictions. Based on these findings, we introduce a divergent-aware NMT framework that uses factors to help NMT recover from the degradation caused by naturally occurring di- vergences, improving both translation quality and model calibration on EN↔FR tasks. 1 Introduction While parallel texts are essential to Neural Ma- chine Translation (NMT), the degree of parallelism varies widely across samples in practice, for rea- sons ranging from noise in the extraction pro- cess (Roziewski and Stokowiec, 2016) to non- literal translations (Zhai et al., 2019b, 2020a). For instance (Figure 1), a French SOURCE could be paired with an exact translation into English (EQ), with a mostly equivalent translation where only a few tokens convey divergent meaning (fine- DIV), or with a semantically unrelated, noisy ref- erence (coarse-DIV). Yet, prior work treats par- allel samples in a binary fashion: coarse-grained divergences are viewed as noise to be excluded from training (Koehn et al., 2018), whilst others are typically regarded as gold-standard equivalent translations. As a result, the impact of fine-grained divergences on NMT remains unclear. This paper aims to understand and mitigate the impact of fine-grained semantic divergences in Figure 1: Equivalent vs. Divergent references on NMT training. Fine-grained divergences (i.e., REF (fine- DIV)) provide an imperfect yet potentially useful signal depending on the time step t. NMT. We first contribute an analysis of how fine- grained divergences in training data affect NMT quality and confidence. Starting from a set of equiv- alent English-French WikiMatrix sentence pairs, we simulate divergences by gradually “corrupting” them with synthetic fine-grained divergences. Fol- lowing Khayrallah and Koehn (2018)—who, in contrast, study the impact of noise on MT—we control for different types of fine-grained semantic divergences and different ratios of equivalent vs. divergent data. Our findings indicate that these im- perfect training references: hurt translation quality (as measured by BLEU and METEOR) once they overwhelm equivalents; output degenerated text more frequently; and increase the uncertainty of models’ predictions. 7237 Based on these findings, we introduce a divergent-aware NMT framework that incorpo- rates information about which tokens are …

Content truncated to 500 words