Selective Attention for Context-aware Neural Machine Translation

Proceedings of NAACL-HLT 2019, pages 3092–3102 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3092 Selective Attention for Context-aware Neural Machine Translation Sameen Maruf†∗ Andr´e F. T. Martins‡ †Faculty of Information Technology, Monash University, VIC, Australia ‡Unbabel & Instituto de Telecomunicac¸˜oes, Lisbon, Portugal †{firstname.lastname}@monash.edu ‡andre.martins@unbabel.com Gholamreza Haffari† Abstract Despite the progress made in sentence-level NMT, current systems still fall short at achiev- ing fluent, good quality translation for a full document. Recent works in context-aware NMT consider only a few previous sentences as context and may not scale to entire docu- ments. To this end, we propose a novel and scalable top-down approach to hierarchical at- tention for context-aware NMT which uses sparse attention to selectively focus on relevant sentences in the document context and then attends to key words in those sentences. We also propose single-level attention approaches based on sentence or word-level information in the context. The document-level context representation, produced from these attention modules, is integrated into the encoder or decoder of the Transformer model depend- ing on whether we use monolingual or bilin- gual context. Our experiments and evaluation on English-German datasets in different doc- ument MT settings show that our selective at- tention approach not only significantly outper- forms context-agnostic baselines but also sur- passes context-aware baselines in most cases. 1 Introduction Neural machine translation has grown immensely in the past few years, from the simplistic RNN- based encoder-decoder models (Sutskever et al., 2014; Bahdanau et al., 2015) to the state-of-the-art Transformer architecture (Vaswani et al., 2017). Most of these models rely on the attention mech- anism as a major component, which involves fo- cusing on different parts of a sequence to com- pute new representations, and has proven to be quite effective in improving the translation quality (Vaswani et al., 2017). However, all of these mod- els share the same inherent problem: the transla- tion is still performed on a sentence-by-sentence ∗Work initiated during an internship at Unbabel. basis, thus ignoring the long-range dependencies which may be useful when it comes to translating discourse phenomena. More recently, context-aware NMT has been gaining significant traction from the MT commu- nity with majority of works coming out in the past two years. Most of these focus on using a few previous sentences as context (Jean et al., 2017; Wang et al., 2017; Tu et al., 2018; Voita et al., 2018; Zhang et al., 2018; Miculicich et al., 2018) and neglect the rest of the document. Only one existing work has endeavoured to consider the full document context (Maruf and Haffari, 2018), thus proposing a more generalised approach to document-level NMT. However, the model is re- strictive as the document-level attention computed is sentence-based and static (computed only once for the sentence being translated). A more recent work (Miculicich et al., 2018) proposes to use a hierarchical attention network (HAN) (Yang et al., 2016) to model the contextual information in a structured manner using word-level and sentence- level abstractions; yet, it uses …

Content truncated to 500 words