Improving Anaphora Resolution in Neural Machine Translation Using Curriculum Learning

Improving Anaphora Resolution in Neural Machine Translation Using Curriculum Learning Dario Stojanovski and Alexander Fraser Center for Information and Language Processing LMU Munich {stojanovski,fraser}@cis.lmu.de Abstract Modeling anaphora resolution is critical for proper pronoun translation in neural machine translation. Recently it has been addressed by context-aware models with varying success. In this work, we pro- pose a carefully designed training curricu- lum that facilitates better anaphora resolu- tion in context-aware NMT. As a baseline, we train context-aware models as was done in previous work. We leverage oracle in- formation specific to anaphora resolution during training. Following the intuition be- hind curriculum learning, we are able to train context-aware models which are im- proved with respect to coreference resolu- tion, even though both the baseline and the improved system have access to exactly the same information at test time. We test our approach using two pronoun-specific eval- uation metrics for MT. 1 Introduction Modeling gender-pronoun agreement and anaphora resolution in machine translation is difficult because most models work on individual sentences. In many cases the antecedent noun is not present in the sentence being translated, but is rather in a preceding sentence. Sentence-external anaphora are a problem in many domains (e.g., consider conversational texts). NMT models can be extended to receive the previous sentences of a document as input. Previous context-aware NMT models include (Jean et al., 2017; Wang © 2019 The authors. This article is licensed under a Creative Commons 4.0 licence, no derivative works, attribution, CC- BY-ND. et al., 2017; Tu et al., 2018; Voita et al., 2018; Stojanovski and Fraser, 2018; Zhang et al., 2018a; Miculicich et al., 2018). Previous work on evaluation has shown that context-aware NMT improves over sentence-level baselines, both in terms of BLEU and in terms of metrics tailored for pronoun evaluation (Bawden et al., 2018; Voita et al., 2018; M¨uller et al., 2018). In this work, we propose a technique for im- proving the ability of context-aware models to han- dle anaphora resolution. The technique is based on curriculum learning (Bengio et al., 2009) which proposes to train neural networks in a similar fash- ion to how humans learn. Curriculum learning is a method that proposes training neural networks by gradually feeding increasingly more complex data instead of training models by randomly showing data samples. We borrow on the intuition behind curriculum learning by initially training models with a form of “training wheels”, where the anaphora relation- ships are made explicit. We take the key idea from previous work, which is to use gold-standard reference pronouns as oracles (Stojanovski and Fraser, 2018). We then gradually remove the or- acles in consecutive fine-tuning steps, until we have a model working without oracle informa- tion. We expect that explicitly showing the ref- erence pronouns in the context will make it easier to model the gender of antecedent nouns and bias the model to do more aggressive anaphora reso- lution when encountering ambiguous pronouns in the source language (the translation of ambiguous pronouns depends on the antecedent). …

Content truncated to 500 words