Modeling Coverage for Neural Machine Translation

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 76–85, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Modeling Coverage for Neural Machine Translation Zhaopeng Tu† Zhengdong Lu† Yang Liu‡ Xiaohua Liu† Hang Li† †Noah’s Ark Lab, Huawei Technologies, Hong Kong {tu.zhaopeng,lu.zhengdong,liuxiaohua3,hangli.hl}@huawei.com ‡Department of Computer Science and Technology, Tsinghua University, Beijing liuyang2011@tsinghua.edu.cn Abstract Attention mechanism has enhanced state- of-the-art Neural Machine Translation (NMT) by jointly learning to align and translate. It tends to ignore past alignment information, however, which often leads to over-translation and under-translation. To address this problem, we propose coverage-based NMT in this paper. We maintain a coverage vector to keep track of the attention history. The coverage vec- tor is fed to the attention model to help ad- just future attention, which lets NMT sys- tem to consider more about untranslated source words. Experiments show that the proposed approach significantly im- proves both translation quality and align- ment quality over standard attention-based NMT.1 1 Introduction The past several years have witnessed the rapid progress of end-to-end Neural Machine Transla- tion (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015). Unlike conventional Statistical Ma- chine Translation (SMT) (Koehn et al., 2003; Chi- ang, 2007), NMT uses a single and large neural network to model the entire translation process. It enjoys the following advantages. First, the use of distributed representations of words can alleviate the curse of dimensionality (Bengio et al., 2003). Second, there is no need to explicitly design fea- tures to capture translation regularities, which is quite difficult in SMT. Instead, NMT is capable of learning representations directly from the training data. Third, Long Short-Term Memory (Hochre- iter and Schmidhuber, 1997) enables NMT to cap- 1Our code is publicly available at https://github. com/tuzhaopeng/NMT-Coverage. ture long-distance reordering, which is a signifi- cant challenge in SMT. NMT has a serious problem, however, namely lack of coverage. In phrase-based SMT (Koehn et al., 2003), a decoder maintains a coverage vec- tor to indicate whether a source word is translated or not. This is important for ensuring that each source word is translated in decoding. The decod- ing process is completed when all source words are “covered” or translated. In NMT, there is no such coverage vector and the decoding process ends only when the end-of-sentence mark is pro- duced. We believe that lacking coverage might result in the following problems in conventional NMT: 1. Over-translation: some words are unneces- sarily translated for multiple times; 2. Under-translation: some words are mistak- enly untranslated. Specifically, in the state-of-the-art attention-based NMT model (Bahdanau et al., 2015), generating a target word heavily depends on the relevant parts of the source sentence, and a source word is in- volved in generation of all target words. As a result, over-translation and under-translation in- evitably happen because of ignoring the “cover- age” of source words (i.e., number of times a source word is translated to a target word). Fig- ure 1(a) shows an example: the Chinese word “gu¯anb`ı” is over translated …

Content truncated to 500 words