A Character-Aware Encoder for Neural Machine Translation

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3063–3070, Osaka, Japan, December 11-17 2016. A Character-Aware Encoder for Neural Machine Translation Zhen Yang, Wei Chen∗, Feng Wang, Bo Xu Institute of Automation, Chinese Academy of Sciences No.95 Zhongguancun East Road {yangzhen2014, wei.chen.media, feng.wang, xubo}@ia.ac.cn Abstract This article proposes a novel character-aware neural machine translation (NMT) model that views the input sequences as sequences of characters rather than words. On the use of row convolution (Amodei et al., 2015), the encoder of the proposed model composes word-level information from the input sequences of characters automatically. Since our model doesn’t rely on the boundaries between each word (as the whitespace boundaries in English), it is also applied to languages without explicit word segmentations (like Chinese). Experimental results on Chinese-English translation tasks show that the proposed character-aware NMT model can achieve comparable translation performance with the traditional word based NMT models. Despite the target side is still word based, the proposed model is able to generate much less unknown words. 1 Introduction Neural machine translation conducts end-to-end translation with a source encoder and a target decoder, producing promising results (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014). With the emerging of the attention-based encoder-decoder model, NMT has achieved comparable even better translation performance with the traditional statistical machine translation (SMT) (Bahdanau et al., 2014; Ranzato et al., 2015; Shen et al., 2015; Tu et al., 2016). The success of NMT lies in its strong ability of composing the global context information. However, as a newly approach, the NMT model has some flaws and limitations that may jeopardize its translation performance (Luong et al., 2014; Sennrich et al., 2015; He et al., 2016). One of the most glaring limitations is that the NMT model is weak in handling the rare and out-of-vocabulary (OOV) words, since the NMT system usually uses the top-N (30000-50000) frequent words in the training corpus and regards other words as unseen words (Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014; Cohn et al., 2016). Two different kinds of approaches have been proposed to handle the OOV problem in NMT: vocabulary-specific approaches and unit-specific approaches. The vocabulary-specific approaches seek to cover more words by using a larger vocabulary (Mnih and Kavukcuoglu, 2013; Cho et al., 2015) or using an identity translation dictionary in a post-processing step (Luong et al., 2014). Intuitively, these approaches can alleviate the OOV problems to a certain extent only if the vocabulary can be expanded large enough. However, these approaches are incapable of solving the OOV problems completely since the vocabulary is always limited. As opposed to vocabulary-specific approaches, the unit-specific approaches try to use more fine- grained processing units than words, like sub-word units (Sennrich et al., 2015) or even character-level units (Ling et al., 2015b; Chung et al., 2016). Regarding the character as the basic processing unit is a new trend in the field of NLP and the character-level models have …

Content truncated to 500 words