Finding the Optimal Vocabulary Size for Neural Machine Translation

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3955–3964 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 3955 Finding the Optimal Vocabulary Size for Neural Machine Translation Thamme Gowda and Jonathan May Information Sciences Institute University of Southern California {tg, jonmay}@isi.edu Abstract We cast neural machine translation (NMT) as a classification task in an autoregressive setting and analyze the limitations of both classification and autoregression components. Classifiers are known to perform better with balanced class distributions during training. Since the Zipfian nature of languages causes imbalanced classes, we explore its effect on NMT. We analyze the effect of various vocab- ulary sizes on NMT performance on multiple languages with many data sizes, and reveal an explanation for why certain vocabulary sizes are better than others.1 1 Introduction Natural language processing (NLP) tasks such as sentiment analysis (Maas et al., 2011; Zhang et al., 2015) and spam detection are modeled as classi- fication tasks, where instances are independently labeled. Tasks such as part-of-speech tagging (Ze- man et al., 2017) and named entity recognition (Tjong Kim Sang and De Meulder, 2003) are ex- amples of structured classification tasks, where in- stance classification is decomposed into a sequence of per-token contextualized labels. We can sim- ilarly cast neural machine translation (NMT), an example of a natural language generation (NLG) task, as a form of structured classification, where an instance label (a translation) is generated as a sequence of contextualized labels, here by an au- toregressor (see Section 2). Since the parameters of modern machine learn- ing (ML) classification models are estimated from training data, whatever biases exist in the training data will affect model performance. Among those biases, class imbalance is a topic of our interest. Class imbalance is said to exist when one or more 1Tools, configurations, system outputs, and analyses are at https://github.com/thammegowda/005-nmt-imbalance classes are not of approximately equal frequency in data. The effect of class imbalance has been extensively studied in several domains where clas- sifiers are used (see Section 6.3). With neural net- works, the imbalanced learning problem is mostly targeted to computer vision tasks; NLP tasks are under-explored (Johnson and Khoshgoftaar, 2019). Word types in natural language models resemble a Zipfian distribution, i.e. in any natural language corpus, we observe that a type’s rank is roughly inversely proportional to its frequency. Thus, a few types are extremely frequent, while most of the rest lie on the long tail of infrequency. Zipfian distributions cause two problems in classifier-based NLG systems: 1. Unseen Vocabulary: Any hidden data set may contain types not seen in the finite set used for training. A sequence drawn from a Zipfian distribution is likely to have a large number of rare types, and these are likely to have not been seen in training. 2. Imbalanced Classes: There are a few ex- tremely frequent types and many infrequent types, causing an extreme imbalance. Such an imbalance, in other domains where classifiers are used, has been known to cause undesired biases and severe …

Content truncated to 500 words