Combining Character and Word Information in Neural Machine Translation Using a Multi-Level Attention

Proceedings of NAACL-HLT 2018, pages 1284–1293 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Combining Character and Word Information in Neural Machine Translation Using a Multi-Level Attention Huadong Chen†, Shujian Huang†∗, David Chiang‡, Xinyu Dai†, Jiajun Chen† †State Key Laboratory for Novel Software Technology, Nanjing University {chenhd,huangsj,daixinyu,chenjj}@nlp.nju.edu.cn ‡Department of Computer Science and Engineering, University of Notre Dame dchiang@nd.edu Abstract Natural language sentences, being hierarchi- cal, can be represented at different levels of granularity, like words, subwords, or charac- ters. But most neural machine translation sys- tems require the sentence to be represented as a sequence at a single level of granularity. It can be difficult to determine which granularity is better for a particular translation task. In this paper, we improve the model by incorporating multiple levels of granularity. Specifically, we propose (1) an encoder with character atten- tion which augments the (sub)word-level rep- resentation with character-level information; (2) a decoder with multiple attentions that en- able the representations from different levels of granularity to control the translation cooper- atively. Experiments on three translation tasks demonstrate that our proposed models out- perform the standard word-based model, the subword-based model and a strong character- based model. 1 Introduction Neural machine translation (NMT) models (Britz et al., 2017) learn to map from source lan- guage sentences to target language sentences via continuous-space intermediate representa- tions. Since word is usually thought of as the ba- sic unit of language communication (Jackendoff, 1992), early NMT systems built these represen- tations starting from the word level (Sutskever et al., 2014; Bahdanau et al., 2015; Cho et al., 2014; Weng et al., 2017). Later systems tried us- ing smaller units such as subwords to address the problem of out-of-vocabulary (OOV) words (Sen- nrich et al., 2016; Wu et al., 2016). Although they obtain reasonable results, these word or sub-word methods still have some poten- tial weaknesses. First, the learned representations ∗ Corresponding author. of (sub)words are based purely on their contexts, but the potentially rich information inside the unit itself is seldom explored. Taking the Chinese word 被打伤 (bei-da-shang) as an example, the three characters in this word are a passive voice marker, “hit” and “wound”, respectively. The meaning of the whole word, “to be wounded”, is fairly com- positional. But this compositionality is ignored if the whole word is treated as a single unit. Secondly, obtaining the word or sub-word boundaries can be non-trivial. For languages like Chinese and Japanese, a word segmentation step is needed, which must usually be trained on la- beled data. For languages like English and Ger- man, word boundaries are easy to detect, but sub- word boundaries need to be learned by methods like BPE. In both cases, the segmentation model is trained only in monolingual data, which may re- sult in units that are not suitable for translation. On the other hand, there have been multiple efforts to build models operating purely at the character level (Ling et al., 2015a; Yang et al., 2016; …

Content truncated to 500 words