Detecting Word Sense Disambiguation Biases in Machine Translation for Model-Agnostic Adversarial Attacks

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 7635–7653, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 7635 Detecting Word Sense Disambiguation Biases in Machine Translation for Model-Agnostic Adversarial Attacks Denis Emelin1, Ivan Titov1, 2, and Rico Sennrich3, 1 1University of Edinburgh, Scotland 2University of Amsterdam, Netherlands 3University of Zurich, Switzerland D.Emelin@sms.ed.ac.uk ititov@inf.ed.ac.uk sennrich@cl.uzh.ch Abstract Word sense disambiguation is a well-known source of translation errors in NMT. We posit that some of the incorrect disambiguation choices are due to models’ over-reliance on dataset artifacts found in training data, specif- ically superficial word co-occurrences, rather than a deeper understanding of the source text. We introduce a method for the predic- tion of disambiguation errors based on statisti- cal data properties, demonstrating its effective- ness across several domains and model types. Moreover, we develop a simple adversarial at- tack strategy that minimally perturbs sentences in order to elicit disambiguation errors to fur- ther probe the robustness of translation models. Our findings indicate that disambiguation ro- bustness varies substantially between domains and that different models trained on the same data are vulnerable to different attacks.1 1 Introduction Consider the sentence John met his wife in the hot spring of 1988. In this context, the polysemous term spring unambiguously refers to the season of a specific year. Its appropriate translation into German would therefore be Fr¨uhling (the season), rather than one of its alternative senses, such as Quelle (the source of a stream). To contemporary machine translation systems, however, this sen- tence presents a non-trivial challenge, with Google Translate (GT) producing the following translation: John traf seine Frau in der heißen Quelle von 1988. Prior studies have indicated that neural machine translation (NMT) models rely heavily on source sentence information when resolving lexical am- biguity (Tang et al., 2019). This suggests that the combined source contexts in which a specific sense of an ambiguous term occurs in the training data 1Experimental codebase available at http://github. com/demelin/detecting_wsd_biases_for_nmt greatly inform the models’ disambiguation deci- sions. Thus, a stronger correlation between the English collocation hot spring and the German translation Quelle, as opposed to Fr¨uhling, in the training corpus may explain this disambiguation error. Indeed, John met his wife in the spring of 1988 is translated correctly by GT. We propose that our motivating example is repre- sentative of a systematic pathology NMT systems have yet to overcome when performing word sense disambiguation (WSD). Specifically, we hypothe- size that translation models learn to disproportion- ately rely on lexical correlations observed in the training data when resolving word sense ambigu- ity. As a result, disambiguation errors are likely to arise when an ambiguous word co-occurs with words that are strongly correlated in the training corpus with a sense that differs from the reference. To test our hypothesis, we evaluate whether dataset artifacts are predictive of disambiguation decisions made in NMT. First, given an ambigu- ous term, we define a strategy for quantifying how much its context biases NMT models towards its different target …

Content truncated to 500 words