Measuring and Increasing Context Usage in Context-Aware Machine Translation

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6467–6478 August 1–6, 2021. ©2021 Association for Computational Linguistics 6467 Measuring and Increasing Context Usage in Context-Aware Machine Translation Patrick Fernandes1,2,3 Kayo Yin1 Graham Neubig1 Andr´e F. T. Martins2,3,4 1Language Technologies Institute, Carnegie Mellon University, Pittsburgh, PA 2Instituto Superior T´ecnico & LUMLIS (Lisbon ELLIS Unit), Lisbon, Portugal 3Instituto de Telecomunicac¸˜oes, Lisbon, Portugal 4Unbabel, Lisbon, Portugal {pfernand, kayoy, gneubig}@cs.cmu.edu andre.t.martins@tecnico.ulisboa.pt Abstract Recent work in neural machine translation has demonstrated both the necessity and feasibil- ity of using inter-sentential context — context from sentences other than those currently be- ing translated. However, while many current methods present model architectures that theo- retically can use this extra context, it is often not clear how much they do actually utilize it at translation time. In this paper, we introduce a new metric, conditional cross-mutual infor- mation, to quantify the usage of context by these models. Using this metric, we measure how much document-level machine translation systems use particular varieties of context. We find that target context is referenced more than source context, and that conditioning on a longer context has a diminishing effect on re- sults. We then introduce a new, simple train- ing method, context-aware word dropout, to increase the usage of context by context-aware models. Experiments show that our method increases context usage and that this reflects on the translation quality according to metrics such as BLEU and COMET, as well as per- formance on anaphoric pronoun resolution and lexical cohesion contrastive datasets.1 1 Introduction While neural machine translation (NMT) is re- ported to have achieved human parity in some do- mains and language pairs (Hassan et al., 2018), these claims seem overly optimistic and no longer hold with document-level evaluation (Toral et al., 2018; L¨aubli et al., 2018). Recent work on context- aware NMT attempts to alleviate this discrepancy by incorporating the surrounding context sentences (in either or both the source and target sides) in the translation system. This can be done by, for ex- ample, feeding context sentences to standard NMT 1https://github.com/neulab/contextual-mt Figure 1: Illustration of how we can measure context usage by a model qMT as the amount of information gained when a model is given the context C and source X vs when the model is only given the X. models (Tiedemann and Scherrer, 2017), using dif- ferent encoders for context (Zhang et al., 2018), having cache-based memories (Tu et al., 2018a), or using models with hierarchical attention mecha- nisms (Miculicich et al., 2018; Maruf et al., 2019a) — more details in §2. While such works report gains in translation quality compared to sentence-level baselines trained on small datasets, recent work has shown that, in more realistic high-resourced scenarios, these systems fail to outperform sim- pler baselines with respect to overall translation accuracy, pronoun translation, or lexical cohesion (Lopes et al., 2020). We hypothesize that one major reason for these lacklustre results is due to …

Content truncated to 500 words