On the Effectiveness of Quasi Character-Level Models for Machine Translation

On the Effectiveness of Quasi Character-Level Models for Machine Translation Salvador Carri´on salcarpo@prhlt.upv.es Francisco Casacuberta fcn@prhlt.upv.es PRHLT Research Center, Universitat Polit`ecnica de Val`encia Abstract Neural Machine Translation (NMT) models often use subword-level vocabularies to deal with rare or unknown words. Although some studies have shown the effectiveness of purely character-based models, these approaches have resulted in highly expensive models in com- putational terms. In this work, we explore the benefits of quasi-character-level models for very low-resource languages and their ability to mitigate the effects of the catastrophic forgetting problem. First, we conduct an empirical study on the efficacy of these models, as a function of the vocabulary and training set size, for a range of languages, domains, and architectures. Next, we study the ability of these models to mitigate the effects of catastrophic forgetting in machine translation. Our work suggests that quasi-character-level models have practically the same generalization capabilities as character-based models but at lower computational costs. Furthermore, they appear to help achieve greater consistency between domains than standard subword-level models, although the catastrophic forgetting problem is not mitigated. 1 Introduction Neural machine translation (NMT) has become the dominant paradigm in the field of ma- chine translation due to the impressive results obtained with the encoder-decoder architec- tures (Sutskever et al., 2014; Cho et al., 2014; Wu et al., 2016; Vaswani et al., 2017) and the approaches proposed to tackle the open vocabulary problem such as subword-based models with byte-fallback or, more recently, token-free models. However, despite these advances, low-resource languages are still problematic as there are many languages that are spoken but not written on the internet (e.g., Tigrinya, Sotho, Tsonga, etc.), and therefore, parallel text mining techniques are either not effective or not applicable at all. In these cases, it is common to use character-based models, since multiple authors have shown that these models usually perform better than (standard) subword- or word-based models in very low-resource settings. Motivated by these ideas, we decided to study whether quasi-character-based vocabularies (defined as a subword-based vocabulary that is one or two orders of magnitude smaller than a standard subword-based vocabulary), had the same advantages as models with character-based vocabularies for low-resource languages, but with much lower computational costs, due to the exponential decrease in the average number of tokens per sentence when merging highly fre- quent character pairs. Furthermore, given that the effects of the catastrophic forgetting problem are strongly re- lated to the vocabulary of the model, we decided to study if these quasi-character-level vocab- ularies had the potential to mitigate them, since these vocabularies are closer to a universal- domain vocabulary (e.g., bytes or chars) than a word- or (standard) subword-based vocabulary. Proceedings of the 15th Biennial Conference of the Association for Machine Translation in the Americas Orlando, USA, September 12-16, 2022. Volume 1: Research Track 131 The contributions of this paper are twofold: • Quasi-character-level models appear to outperform character-based models in terms of performance, while offering practically the same generalization capabilities at much lower computational costs. • …

Content truncated to 500 words