How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation?

How Effective is Byte Pair Encoding for Out-Of-Vocabulary Words in Neural Machine Translation? Ali Araabi a.araabi@uva.nl Christof Monz c.monz@uva.nl Vlad Niculae v.niculae@uva.nl Informatics Institute, University of Amsterdam, Amsterdam, The Netherlands Abstract Neural Machine Translation (NMT) is an open vocabulary problem. As a result, dealing with the words not occurring during training (a.k.a. out-of-vocabulary (OOV) words) have long been a fundamental challenge for NMT systems. The predominant method to tackle this problem is Byte Pair Encoding (BPE) which splits words, including OOV words, into sub-word segments. BPE has achieved impressive results for a wide range of translation tasks in terms of automatic evaluation metrics. While it is often assumed that by using BPE, NMT systems are capable of handling OOV words, the effectiveness of BPE in translating OOV words has not been explicitly measured. In this paper, we study to what extent BPE is successful in translating OOV words at the word-level. We analyze the translation quality of OOV words based on word type, number of segments, cross-attention weights, and the frequency of segment n-grams in the training data. Our experiments show that while careful BPE settings seem to be fairly useful in translating OOV words across datasets, a considerable percentage of OOV words are translated incorrectly. Furthermore, we highlight the slightly higher effectiveness of BPE in translating OOV words for special cases, such as named-entities and when the languages involved are linguistically close to each other. 1 Introduction One of the key challenges of neural machine translation (NMT) (Sutskever et al., 2014; Bahdanau et al., 2015) is vocabulary sparsity; irrespective of the amount of data available for training. As a consequence, all NMT models suffer from out-of-vocabulary (OOV) words. Accordingly, a significant proportion of sentences in the test set have OOV words. Even with millions of sentence pairs in training data,1 15% of test sentences contain OOV words, while with limited training data,2 OOV words appear in more than 60% of the test sentences. Earlier approaches to tackling the OOV problem include using a very large vocabulary (Jean et al., 2015), backing off to a dictionary look-up (Luong et al., 2015), and copying OOV words from source to the target sentence (Gülçehre et al., 2016). However, most recent approaches are based on splitting the words into smaller units and can be divided into: language-specific approaches (Smit et al., 2014; Huck et al., 2017), language-agnostic approaches (Sennrich et al., 1Russian-English from WMT 2Kazakh-English from WMT Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 117 2016; Kudo, 2018; Kudo and Richardson, 2018; Costa-jussà and Fonollosa, 2016; Cherry et al., 2018), and hybrid approaches (Huck et al., 2017; Banerjee and Bhattacharyya, 2018; Pan et al., 2020) which inject linguistic information into language-agnostic methods. Nowadays, the mainstream approach to address the open-vocabulary challenge in the context of NMT is Byte Pair Encoding (BPE Sennrich et al., 2016), due to its simplicity, applicability to a wide range of languages, …

Content truncated to 500 words