Contrastive Learning for Context-aware Neural Machine Translation Using Coreference Information

Proceedings of the Sixth Conference on Machine Translation (WMT), pages 1135–1144 November 10–11, 2021. ©2021 Association for Computational Linguistics 1135 Contrastive Learning for Context-aware Neural Machine Translation Using Coreference Information Yongkeun Hwang1, Hyungu Yun1, Kyomin Jung1,2 1Dept. of Electrical and Computer Engineering, Seoul National University, Seoul, Korea 2Automation and Systems Research Institute, Seoul National University, Seoul, Korea {wangcho2k, youaredead, kjung}@snu.ac.kr Abstract Context-aware neural machine translation (NMT) incorporates contextual information of surrounding texts, that can improve the transla- tion quality of document-level machine trans- lation. Many existing works on context-aware NMT have focused on developing new model architectures for incorporating additional con- texts and have shown some promising results. However, most existing works rely on cross- entropy loss, resulting in limited use of con- textual information. In this paper, we propose CorefCL, a novel data augmentation and con- trastive learning scheme based on coreference between the source and contextual sentences. By corrupting automatically detected coref- erence mentions in the contextual sentence, CorefCL can train the model to be sensitive to coreference inconsistency. We experimented with our method on common context-aware NMT models and two document-level transla- tion tasks. In the experiments, our method con- sistently improved BLEU of compared models on English-German and English-Korean tasks. We also show that our method significantly im- proves coreference resolution in the English- German contrastive test suite. 1 Introduction Neural machine translation (NMT) has achieved impressive performances on translation quality, due to the introduction of novel deep neural net- work (DNN) architectures such as encoder-decoder model (Cho et al., 2014; Sutskever et al., 2014), and self-attentional networks like Transformer (Vaswani et al., 2017). The state-of-the-art NMT systems are now even comparable with human translators in sentence-level performance. However, there are a number of issues on document-level translation (Läubli et al., 2018). These include pronoun resolution across sentences (Guillou et al., 2018), which needs cross-sentential contexts. To incorporate such document-level con- textual information, several methods for context- aware NMT have been recently proposed. Many of the works have focused on introducing new model architectures like multi-encoder models (Voita et al., 2018) for encompassing contextual texts of the source language. These works have shown significant improvement in addressing dis- course phenomena such as anaphora resolution mentioned above, as well as moderate improve- ments in overall translation quality (Lopes et al., 2020). Despite some promising results, most of the ex- isting works have trained the model by minimizing cross-entropy loss, making the model rather exploit contextual information implicitly such as a form of regularization (Kim et al., 2019; Li et al., 2020). Data augmentation for context-aware NMT is also an important issue, despite that recent works have focused on back-translation (Huo et al., 2020). In this paper, we propose a Coreference-based Contrastive Learning for context-aware NMT (CorefCL), a novel data augmentation and con- trastive learning scheme leveraging coreference in- formation. Cross-sentential coreference between the source and target sentence can be a good source of training signal for context-aware NMT since it occurs when one or more expressions refer …

Content truncated to 500 words