Feature Decay Algorithms for Neural Machine Translation

Feature Decay Algorithms for Neural Machine Translation Alberto Poncelas, Gideon Maillette de Buy Wenniger, Andy Way ADAPT Centre, School of Computing, Dublin City University, Dublin, Ireland {firstname.lastname}@adaptcentre.ie Abstract Neural Machine Translation (NMT) systems require a lot of data to be com- petitive. For this reason, data selec- tion techniques are used only for fine- tuning systems that have been trained with larger amounts of data. In this work we aim to use Feature Decay Al- gorithms (FDA) data selection tech- niques not only to fine-tune a sys- tem but also to build a complete sys- tem with less data. Our findings re- veal that it is possible to find a subset of sentence pairs, that outperforms by 1.11 BLEU points the full training cor- pus, when used for training a German- English NMT system . 1 Introduction In Statistical Machine Translation (SMT) it has been shown that having more data does not always lead to better results [Ozdowska and Way, 2009]. In fact, performance can increase by limiting the training data to a smaller but more relevant set [Eetemadi et al., 2015]. Neural Machine Translation (NMT) c⃝ 2018 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, at- tribution, CC-BY-ND. models in contrast are data-hungry, and per- form better only with large amounts of avail- able training data, in some cases underper- forming SMT when low amounts of data are available [ ¨Ostling and Tiedemann, 2017; Dowling et al., 2018]. However, the amount of training data required to make NMT work really well depends a lot on the domain of the training data and test set, and possibly also how similar they are. For certain training do- mains such as TED talks [Bentivogli et al., 2016] it has already been shown that even with fairly limited training sizes NMT can already outperform SMT by a large margin. Larger training sets also introduce noise and require models to cover a larger number of words, whereas for practical reasons the vo- cabulary cannot be arbitrarily increased to fa- cilitate these extra words. Consequently, train- ing material that is not relevant for the test set risks wasting limited entries in the vocabulary on source words that are not relevant to the test domain. This is why domain adaptation has proven to be useful in NMT [Chu et al., 2017] by tailoring a model towards in-domain data. While traditional Machine Translation (MT) approaches perform an inductive learning (i.e. learn a model from translated sentences in order to predict unseen ex- amples), transductive learning approaches P´erez-Ortiz, S´anchez-Mart´ınez, Espl`a-Gomis, Popovi´c, Rico, Martins, Van den Bogaert, Forcada (eds.) Proceedings of the 21st Annual Conference of the European Association for Machine Translation, p. 239–248 Alacant, Spain, May 2018. aim to identify the best training instance to predict the test set [Bianchini et al., 2016]. Models trained with sentences retrieved by transductive learning methods are tailored towards the test set. This is similar to the way many domain-adaptation methods adapt the training material …

Content truncated to 500 words