Learning to Remember Translation History with a Continuous Cache

Learning to Remember Translation History with a Continuous Cache Zhaopeng Tu Tencent AI Lab zptu@tencent.com Yang Liu Tsinghua University liuyang2011@tsinghua.edu.cn Shuming Shi Tencent AI Lab shumingshi@tencent.com Tong Zhang Tencent AI Lab bradymzhang@tencent.com Abstract Existing neural machine translation (NMT) models generally translate sentences in isola- tion, missing the opportunity to take advan- tage of document-level information. In this work, we propose to augment NMT models with a very light-weight cache-like memory network, which stores recent hidden represen- tations as translation history. The probabil- ity distribution over generated words is up- dated online depending on the translation his- tory retrieved from the memory, endowing NMT models with the capability to dynami- cally adapt over time. Experiments on mul- tiple domains with different topics and styles show the effectiveness of the proposed ap- proach with negligible impact on the compu- tational cost. 1 Introduction Neural machine translation (NMT) has advanced the state of the art in recent years (Kalchbrenner et al., 2014; Cho et al., 2014; Sutskever et al., 2014; Bah- danau et al., 2015). However, existing models gen- erally treat documents as a list of independent sen- tence pairs and ignore cross-sentence information, which leads to translation inconsistency and ambi- guity arising from a single source sentence. There have been few recent attempts to model cross-sentence context for NMT: Wang et al. (2017a) use a hierarchical RNN to summarize the previous K source sentences, while Jean et al. (2017) use an additional set of an encoder and at- tention model to dynamically select part of the pre- vious source sentence. While these approaches have proven their ability to represent cross-sentence con- text, they generate the context from discrete lexi- cons, thus would cause errors propagated from gen- erated translations. Accordingly, they only take into account source sentences but fail to make use of target-side information.1 Another potential lim- itation is that they are computationally expensive, which limits the scale of cross-sentence context. In this work, we propose a very light-weight alter- native that can both cover large-scale cross-sentence context as well as exploit bilingual translation his- tory. Our work is inspired by recent successes of memory-augmented neural networks on multi- ple NLP tasks (Weston et al., 2015; Sukhbaatar et al., 2015; Miller et al., 2016; Gu et al., 2018), es- pecially the efficient cache-like memory networks for language modeling (Grave et al., 2017; Daniluk et al., 2017). Specifically, the proposed approach augments NMT models with a continuous cache (CACHE), which stores recent hidden representa- tions as history context. By minimizing the compu- tation burden of the cache-like memory, we are able to use larger memory and scale to longer translation history. Since we leverage internal representations instead of output words, our approach is more ro- bust to the error propagation problem, and thus can incorporate useful target-side context. Experimental results show that the proposed approach significantly and consistently improves translation performance over a strong NMT baseline on multiple domains with different topics and styles. We found the introduced cache is able to …

Content truncated to 500 words