Naive Regularizers for Low-Resource Neural Machine Translation

Proceedings of Recent Advances in Natural Language Processing, pages 102–111, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_013 102 Naive Regularizers for Low-Resource Neural Machine Translation Meriem Beloucif1, Ana Valeria Gonzalez2, Marcel Bollmann2, and Anders Søgaard2 1Language Technology Group, Universität Hamburg, Hamburg, Germany 2Dpt. of Computer Science, University of Copenhagen, Copenhagen, Denmark Abstract Neural machine translation models have little inductive bias, which can be a disad- vantage in low-resource scenarios. They require large volumes of data and often perform poorly when limited data is avail- able. We show that using naive regular- ization methods, based on sentence length, punctuation and word frequencies, to pe- nalize translations that are very different from the input sentences, consistently im- proves the translation quality across mul- tiple low-resource languages. We ex- periment with 12 language pairs, vary- ing the training data size between 17k to 230k sentence pairs. Our best regularizer achieves an average increase of 1.5 BLEU score and 1.0 TER score across all the language pairs. For example, we achieve a BLEU score of 26.70 on the IWSLT15 English–Vietnamese translation task sim- ply by using relative differences in punc- tuation as a regularizer. 1 Introduction One of the major challenges when training neu- ral networks is overfitting. Overfitting is what happens when a neural network in part memo- rizes the training data rather than learning to gen- eralize from it. To prevent this, neural machine translation (NMT) models are typically trained with an L1 or L2 penalty, dropout, momentum, or other general-purpose regularizers. General- purpose regularizers and large volumes of training data have enabled us to train flexible, expressive neural machine translation architectures that have provided a new state of the art in machine transla- tion. For low-resource language pairs, however, where large volumes of training data are not avail- able, neural machine translation has come with diminishing returns (Koehn and Knowles, 2017). The general-purpose regularizers do not provide enough inductive bias to enable generalization, it seems. This is an area of active research, and other work has explored multi-task learning (Firat et al., 2016; Dong et al., 2015), zero-shot learning (John- son et al., 2016), and unsupervised machine trans- lation (Gehring et al., 2017) to resolve the data bottleneck. In this paper, we consider a fully com- plementary, but much simpler alternative: naive, linguistically motivated regularizers that penalize the output sentences of translation models depart- ing heavily from simple characteristics of the input sentences. The proposed regularizers are based on three surface properties of sentences: their length (mea- sured as number of tokens), their amount of punctuation (measured as number of punctuation signs), and the frequencies of their words (as mea- sured on external corpora). While there are lan- guages that do not make use of punctuation (e.g., Lao and Thai), in general these three properties are roughly preserved across translations into most languages. If we translate a sentence such as (1), for example: (1) That dog is a Chinook. it is relatively safe to assume that a good trans- lation will be …

Content truncated to 500 words