Improving Neural Machine Translation by Incorporating Hierarchical Subword Features

Proceedings of the 27th International Conference on Computational Linguistics, pages 618–629 Santa Fe, New Mexico, USA, August 20-26, 2018. 618 Improving Neural Machine Translation by Incorporating Hierarchical Subword Features Makoto Morishita, Jun Suzuki∗and Masaaki Nagata NTT Communication Science Laboratories, NTT Corporation, Japan {morishita.makoto, nagata.masaaki}@lab.ntt.co.jp jun.suzuki@ecei.tohoku.ac.jp Abstract This paper focuses on subword-based Neural Machine Translation (NMT). We hypothesize that in the NMT model, the appropriate subword units for the following three modules (layers) can differ: (1) the encoder embedding layer, (2) the decoder embedding layer, and (3) the decoder output layer. We find the subword based on Sennrich et al. (2016) has a feature that a large vocab- ulary is a superset of a small vocabulary and modify the NMT model enables the incorporation of several different subword units in a single embedding layer. We refer these small subword fea- tures as hierarchical subword features. To empirically investigate our assumption, we compare the performance of several different subword units and hierarchical subword features for both the encoder and decoder embedding layers. We confirmed that incorporating hierarchical subword features in the encoder consistently improves BLEU scores on the IWSLT evaluation datasets. Title and Abstract in Japanese 階層的部分単語特徴を用いたニューラル機械翻訳 本稿では、部分単語 (subword) を用いたニューラル機械翻訳 (Neural Machine Translation, NMT) に着目する。NMT モデルでは、エンコーダの埋め込み層、デコーダの埋め込み層お よびデコーダの出力層の 3 箇所で部分単語が用いられるが、それぞれの層で適切な部分単 語単位は異なるという仮説を立てた。我々は、Sennrich et al. (2016) に基づく部分単語は、 大きな語彙集合が小さい語彙集合を必ず包含するという特徴を利用して、複数の異なる部 分単語列を同時に一つの埋め込み層として扱えるよう NMT モデルを改良する。以降、こ の小さな語彙集合特徴を階層的部分単語特徴と呼ぶ。本仮説を検証するために、様々な部 分単語単位や階層的部分単語特徴をエンコーダ・デコーダの埋め込み層に適用して、その 精度の変化を確認する。IWSLT 評価セットを用いた実験により、エンコーダ側で階層的な 部分単語を用いたモデルは BLEU スコアが一貫して向上することが確認できた。 1 Introduction The approach of end-to-end Neural Machine Translation (NMT) continues to make rapid progress. A simple encoder-decoder model was proposed by Sutskever et al. (2014), and an attentional mechanism was added to better exploit the encoder-side information (Luong et al., 2015; Bahdanau et al., 2015). Compared to traditional Statistical Machine Translation (SMT), NMT has relatively simple architecture, which only uses a large single neural network, but its accuracy surpasses SMT (Junczys-Dowmunt et al., 2016). A conventional NMT uses a limited vocabulary and a decoder generates a “word” in the vocabulary at each time step, but a problem occurs when it encounters an out-of-vocabulary word. Since NMT cannot correctly encode and generate such out-of-vocabulary words, the task performance is degraded. To solve this problem, Sennrich et al. (2016) proposed a method that expresses a word by combining “subwords.” ∗His current affiliation is Tohoku University. This work is licensed under a Creative Commons Attribution 4.0 International License. License details: http:// creativecommons.org/licenses/by/4.0/ 619 A subword is a fraction of a word, determined by Byte Pair Encoding (BPE) operations. By the BPE operation, a word that appears frequently can be one unit, and rare or uncommon words can be expressed by the combination of several subword units. Thus we can express any words by the combination of small subword vocabularies to alleviate the out-of-vocabulary problem. Several similar works exist to make subword units (e.g., (Schuster and Nakajima, 2012; Kudo, 2018)), but in the following, we denote the units segmented by BPE as subword units unless otherwise noted. The primary reason why we use subword units is to generate rare or unknown words on the decoder side. In other words, our …

Content truncated to 500 words