Fully Character-Level Neural Machine Translation without Explicit Segmentation

Fully Character-Level Neural Machine Translation without Explicit Segmentation Jason Lee∗ ETH Z¨urich jasonlee@inf.ethz.ch Kyunghyun Cho New York University kyunghyun.cho@nyu.edu Thomas Hofmann ETH Z¨urich thomas.hofmann@inf.ethz.ch Abstract Most existing machine translation systems op- erate at the level of words, relying on ex- plicit segmentation to extract tokens. We in- troduce a neural machine translation (NMT) model that maps a source character sequence to a target character sequence without any seg- mentation. We employ a character-level con- volutional network with max-pooling at the encoder to reduce the length of source rep- resentation, allowing the model to be trained at a speed comparable to subword-level mod- els while capturing local regularities. Our character-to-character model outperforms a recently proposed baseline with a subword- level encoder on WMT’15 DE-EN and CS- EN, and gives comparable performance on FI- EN and RU-EN. We then demonstrate that it is possible to share a single character- level encoder across multiple languages by training a model on a many-to-one transla- tion task. In this multilingual setting, the character-level encoder significantly outper- forms the subword-level encoder on all the language pairs. We observe that on CS-EN, FI-EN and RU-EN, the quality of the multilin- gual character-level translation even surpasses the models specifically trained on that lan- guage pair alone, both in terms of the BLEU score and human judgment. 1 Introduction Nearly all previous work in machine translation has been at the level of words. Aside from our intu- ∗The majority of this work was completed while the author was visiting New York University. itive understanding of word as a basic unit of mean- ing (Jackendoff, 1992), one reason behind this is that sequences are significantly longer when rep- resented in characters, compounding the problem of data sparsity and modeling long-range depen- dencies. This has driven NMT research to be al- most exclusively word-level (Bahdanau et al., 2015; Sutskever et al., 2014). Despite their remarkable success, word-level NMT models suffer from several major weaknesses. For one, they are unable to model rare, out-of- vocabulary words, making them limited in translat- ing languages with rich morphology such as Czech, Finnish and Turkish. If one uses a large vocabulary to combat this (Jean et al., 2015), the complexity of training and decoding grows linearly with respect to the target vocabulary size, leading to a vicious cycle. To address this, we present a fully character-level NMT model that maps a character sequence in a source language to a character sequence in a target language. We show that our model outperforms a baseline with a subword-level encoder on DE-EN and CS-EN, and achieves a comparable result on FI-EN and RU-EN. A purely character-level NMT model with a basic encoder was proposed as a base- line by Luong and Manning (2016), but training it was prohibitively slow. We were able to train our model at a reasonable speed by drastically reducing the length of source sentence representation using a stack of convolutional, pooling and highway layers. One advantage of character-level models is that they are better …

Content truncated to 500 words