Coverage Embedding Models for Neural Machine Translation

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 955–960, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Coverage Embedding Models for Neural Machine Translation Haitao Mi Baskaran Sankaran Zhiguo Wang Abe Ittycheriah∗ T.J. Watson Research Center IBM 1101 Kitchawan Rd, Yorktown Heights, NY 10598 {hmi, bsankara, zhigwang, abei}@us.ibm.com Abstract In this paper, we enhance the attention-based neural machine translation (NMT) by adding explicit coverage embedding models to alle- viate issues of repeating and dropping trans- lations in NMT. For each source word, our model starts with a full coverage embedding vector to track the coverage status, and then keeps updating it with neural networks as the translation goes. Experiments on the large-scale Chinese-to-English task show that our enhanced model improves the translation quality significantly on various test sets over the strong large vocabulary NMT system. 1 Introduction Neural machine translation (NMT) has gained pop- ularity in recent years (e.g. (Bahdanau et al., 2014; Jean et al., 2015; Luong et al., 2015; Mi et al., 2016b; Li et al., 2016)), especially for the attention- based models of Bahdanau et al. (2014). The at- tention at each time step shows which source word the model should focus on to predict the next tar- get word. However, the attention in each step only looks at the previous hidden state and the previous target word, there is no history or coverage infor- mation typically for each source word. As a result, this kind of model suffers from issues of repeating or dropping translations. The traditional statistical machine translation (SMT) systems (e.g. (Koehn, 2004)) address the above issues by employing a source side “cover- age vector” for each sentence to indicate explicitly which words have been translated, which parts have not yet. A coverage vector starts with all zeros, meaning no word has been translated. If a source word at position j got translated, the coverage vector sets position j as 1, and they won’t use this source ∗Work done while at IBM. To contact Abe, aitty- cheriah@google.com. word in future translation. This mechanism avoids the repeating or dropping translation problems. However, it is not easy to adapt the “coverage vec- tor” to NMT directly, as attentions are soft probabili- ties, not 0 or 1. And SMT approaches handle one-to- many fertilities by using phrases or hiero rules (pre- dict several words in one step), while NMT systems only predict one word at each step. In order to alleviate all those issues, we borrow the basic idea of “coverage vector”, and introduce a coverage embedding vector for each source word. We keep updating those embedding vectors at each translation step, and use those vectors to track the coverage information. Here is a brief description of our approach. At the beginning of translation, we start from a full cover- age embedding vector for each source word. This is different from the “coverage vector” in SMT in following two aspects: • each source word has its own coverage embed- ding …

Content truncated to 500 words