Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1527–1537, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1527 Enhancing Context Modeling with a Query-Guided Capsule Network for Document-level Translation Zhengxin Yang123, Jinchao Zhang3, Fandong Meng3 Shuhao Gu12, Yang Feng12⋆, Jie Zhou3 1 University of Chinese Academy of Sciences 2 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 3 Pattern Recognition Center, WeChat AI, Tencent Inc, China {yangzhengxin17z,gushuhao17g,fengyang}@ict.ac.cn {dayerzhang,fandongmeng,withtomzhou}@tencent.com Abstract Context modeling is essential to gener- ate coherent and consistent translation for Document-level Neural Machine Translations. The widely used method for document-level translation usually compresses the context in- formation into a representation via hierarchi- cal attention networks. However, this method neither considers the relationship between context words nor distinguishes the roles of context words. To address this problem, we propose a query-guided capsule networks to cluster context information into different perspectives from which the target transla- tion may concern. Experiment results show that our method can significantly outperform strong baselines on multiple data sets of dif- ferent domains. 1 Introduction The encoder-decoder based Neural machine trans- lation (NMT) models (Sutskever et al., 2014; Bah- danau et al., 2014; Wu et al., 2016; Vaswani et al., 2017; Zhang et al., 2019) have made great pro- gresses and drawn much attention in recent years. In practical applications, NMT systems are often fed with a document-level input which requires reference resolution, the consistency of tenses and noun expressions and so on. Many researchers have proven that contextual information is essen- tial to generate coherent and consistent translation for document-level translation (Hardmeier, 2012; Meyer and Webber, 2013; Sim Smith, 2017; Jean et al., 2017; Maruf and Haffari, 2018; Miculicich et al., 2018; Zhang et al., 2018a; Voita et al., 2018; Wang et al., 2017; Tu et al., 2018; Maruf et al., 2019). Despite the great success of the above mod- els, they are designed for sentence-level transla- Joint work with Pattern Recognition Center, WeChat AI, Tencent Inc, China. ⋆ Corresponding Author tion tasks and exclude contextual information in the model architecture. On these grounds, the widely used hierarchi- cal attention networks (HAN) was proposed to in- tegrate contextual information in document-level translation (Miculicich et al., 2018). In this method, the context sentences are considered in the form of hierarchical attentions retrieved by the current generation. That is, it utilizes a word- level attention to represent a sentence and then a sentence-level attention to represent all the in- volved context. In this way, the final attention representation has to encode all the information needed for coherent and consistent translation, in- cluding reference information, tenses, expressions and so on. To get the multi-perspective informa- tion, it is necessary to distinguish the role of each context word and model their relationship espe- cially when one context word could take on mul- tiple roles (Zhang et al., 2018b). However, this is difficult to realize …

Content truncated to 500 words