ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation

Proceedings of the 28th International Conference on Computational Linguistics, pages 4732–4749 Barcelona, Spain (Online), December 8-13, 2020 4732 ContraCAT: Contrastive Coreference Analytical Templates for Machine Translation Dario Stojanovski†∗, Benno Krojer†∗, Denis Peskov‡∗, Alexander Fraser† †Center for Information and Language Processing, LMU Munich ‡Computer Science, University of Maryland {stojanovski,fraser}@cis.lmu.de, benno.krojer@gmail.com, dpeskov@cs.umd.edu Abstract Recent high scores on pronoun translation using context-aware neural machine translation have suggested that current approaches work well. ContraPro is a notable example of a contrastive challenge set for English→German pronoun translation. The high scores achieved by transformer models may suggest that they are able to effectively model the complicated set of inferences required to carry out pronoun translation. This entails the ability to determine which entities could be referred to, identify which entity a source- language pronoun refers to (if any), and access the target-language grammatical gender for that entity. We first show through a series of targeted adversarial attacks that in fact current approaches are not able to model all of this information well. Inserting small amounts of distracting information is enough to strongly reduce scores, which should not be the case. We then create a new template test set Contracat, designed to individually assess the ability to handle the specific steps necessary for successful pronoun translation. Our analyses show that current approaches to context-aware nmt rely on a set of surface heuristics, which break down when translations require real reasoning. We also propose an approach for augmenting the training data, with some improvements. 1 Introduction Machine translation is a complex task which requires diverse linguistic knowledge. The seem- ingly straightforward translation of the English pronoun it into German requires knowledge at the syntactic, discourse and world knowledge levels for proper pronoun coreference resolution (cr). The German third person pronoun can have three genders, determined by its antecedent: masculine (er), feminine (sie) and neuter (es). Previous work (Hardmeier and Federico, 2010; Miculicich Werlen and Popescu-Belis, 2017; Müller et al., 2018) proposed evaluation methods for pronoun translation. This has been of special interest in context-aware nmt models that are capable of using discourse-level information. Despite promising results (Bawden et al., 2018; Müller et al., 2018; Lopes et al., 2020), the question remains: Are transformers (Vaswani et al., 2017) truly learning this task, or are they exploiting simple heuristics to make a coreference prediction? To empirically answer this question, we extend ContraPro (Müller et al., 2018)—a contrastive challenge set for automatic English→German pronoun translation evaluation—by making small adversarial changes in the contextual sentences. Our adversarial attacks on ContraPro show that context-aware transformer nmt models can easily be misled by simple and unimportant changes to the input. However, interpreting the results obtained from adversarial attacks can be difficult. The results indicate that nmt uses brittle heuristics to solve cr, but it is not clear what those heuristics are. In general, it is challenging to design attacks based on modifying ContraPro that can test specific phenomena that may be of interest. ∗Equal contribution. This work is licensed under a Creative Commons …

Content truncated to 500 words