An Evaluation of Two Vocabulary Reduction Methods for Neural Machine Translation

An Evaluation of Two Vocabulary Reduction Methods for Neural Machine Translation Duygu Ataman ataman@fbk.eu Fondazione Bruno Kessler, Trento, Italy University of Trento, Trento, Italy Marcello Federico federico@fbk.eu MMT Srl, Trento, Italy Fondazione Bruno Kessler, Trento, Italy Abstract Neural machine translation (NMT) models are conventionally trained with fixed-size vocabu- laries to control the computational complexity and the quality of the learned word represen- tations. This, however, limits the accuracy and the generalization capability of the models, especially for morphologically-rich languages, which usually have very sparse vocabularies containing rare inflected or derivated word forms. Some studies tried to overcome this prob- lem by segmenting words into subword level representations and modeling translation at this level. However, recent findings have shown that if these methods interrupt the word struc- ture during segmentation, they might cause semantic or syntactic losses and lead to generat- ing inaccurate translations. In order to investigate this phenomenon, we present an extensive evaluation of two unsupervised vocabulary reduction methods in NMT. The first is the well- known byte-pair-encoding (BPE), a statistical subword segmentation method, whereas the sec- ond is linguistically-motivated vocabulary reduction (LMVR), a segmentation method which also considers morphological properties of subwords. We compare both approaches on ten translation directions involving English and five other languages (Arabic, Czech, German, Ital- ian and Turkish), each representing a distinct language family and morphological typology. LMVR obtains significantly better performance in most languages, showing gains proportional to the sparseness of the vocabulary and the morphological complexity of the tested language. 1 Introduction Neural machine translation (NMT) has provided significant improvements to the state-of-the- art in machine translation (Bentivogli et al., 2016). However, it has also brought quite a few practical issues. A very important one of these is the low accuracy in translating rare words, caused by two of the main properties of the model. The first is related to the requirement of observing many examples of a word until its internal representation becomes accurate, and the second is due to the difficulty of handling large vocabularies, as this has an impact on the computational complexity of the model. Current implementations of NMT models require long training time and large memory space due to the high number of parameters to optimize. Hence, even with the most advanced machinery, deploying networks that can learn reliable representa- tions for all words observed in the training corpus becomes practically impossible. In order to control the model complexity and the quality of the word representations, a straightforward ap- proach is to fix the vocabularies to a maximum size, e.g. 100,000 lexical units, prior to training. Proceedings of AMTA 2018, vol. 1: MT Research Track Boston, March 17 - 21, 2018 | Page 97 Clearly, a word can only be translated if an exact match of it is found in the vocabulary. This requirement leads to critical restrictions in translating morphologically-rich languages, where the word vocabulary tends to be very large and sparse. For example, in our case study, despite the relatively small size of our …

Content truncated to 500 words