Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 49–60 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64006 Coreference and Coherence in Neural Machine Translation: A Study Using Oracle Experiments Dario Stojanovski Alexander Fraser Center for Information and Language Processing LMU Munich {stojanovski,fraser}@cis.lmu.de Abstract Cross-sentence context can provide valuable information in Machine Translation and is crit- ical for translation of anaphoric pronouns and for providing consistent translations. In this paper, we devise simple oracle experiments targeting coreference and coherence. Oracles are an easy way to evaluate the effect of dif- ferent discourse-level phenomena in NMT us- ing BLEU and eliminate the necessity to man- ually define challenge sets for this purpose. We propose two context-aware NMT mod- els and compare them against models work- ing on a concatenation of consecutive sen- tences. Concatenation models perform better, but are computationally expensive. We show that NMT models taking advantage of context oracle signals can achieve considerable gains in BLEU, of up to 7.02 BLEU for corefer- ence and 1.89 BLEU for coherence on subti- tles translation. Access to strong signals al- lows us to make clear comparisons between context-aware models. 1 Introduction Neural Machine Translation (NMT) (Bahdanau et al., 2015) is a state-of-the-art approach to MT. Standard NMT models translate an input language sentence to an output language sentence, and do not take into account discourse-level phenomena. Cross-sentence context has already proven useful for language modeling (Ji et al., 2015; Wang and Cho, 2016) and dialogue systems (Serban et al., 2016). It has also been of interest in Statistical Machine Translation (SMT) research (Hardmeier, 2012; Hardmeier et al., 2013; Carpuat and Simard, 2012), and NMT research (Wang et al., 2017; Jean et al., 2017; Tiedemann and Scherrer, 2017; Baw- den et al., 2018; Tu et al., 2017; Voita et al., 2018). Two important discourse phenomena for MT are coreference and coherence. Pronominal coref- erence relates to the issue of translating anaphoric pronouns and is tackled in several works (Guillou, 2016; Hardmeier and Federico, 2010; Le Nagard and Koehn, 2010) and is the central motivation for the DiscoMT shared task on cross-lingual pronoun prediction (Lo´aiciga et al., 2017). Coherence on the other hand, is important for producing consis- tent and coherent translations throughout a docu- ment, especially for domain-specific terminology (Carpuat, 2009; Ture et al., 2012; Gonzales et al., 2017) and it is helpful to properly disambiguate polysemous words. Modeling discourse-level phe- nomena for MT is a challenging endeavor because of difficulties in acquiring relevant linguistic sig- nals. Measuring the effect of discourse-level phe- nomena with automatic metrics such as BLEU is also difficult as pointed out by Hardmeier (2012). In this paper, we address these issues by propos- ing several oracle experimental setups for eval- uating the effect of coreference resolution (CR) and coherence in MT. Oracle experiments provide strong linguistic signals that enable strongly vis- ible effects on BLEU scores, thus alleviating the difficulty of using BLEU to evaluate discourse- …

Content truncated to 500 words