Data Selection with Feature Decay Algorithms Using an Approximated Target Side

Data Selection with Feature Decay Algorithms Using an Approximated Target Side Alberto Poncelas, Gideon Maillette de Buy Wenniger, Andy Way ADAPT Centre, School of Computing, Dublin City University, Dublin, Ireland {firstname.lastname}@adaptcentre.ie Abstract Data selection techniques applied to neural machine trans- lation (NMT) aim to increase the performance of a model by retrieving a subset of sentences for use as training data. One of the possible data selection techniques are trans- ductive learning methods, which select the data based on the test set, i.e. the document to be translated. A limitation of these methods to date is that using the source-side test set does not by itself guarantee that sentences are selected with correct translations, or translations that are suitable given the test-set domain. Some corpora, such as subtitle corpora, may contain parallel sentences with inaccurate translations caused by localization or length restrictions. In order to try to fix this problem, in this paper we pro- pose to use an approximated target-side in addition to the source-side when selecting suitable sentence-pairs for train- ing a model. This approximated target-side is built by pre- translating the source-side. In this work, we explore the performance of this general idea for one specific data selection approach called Feature Decay Algorithms (FDA). We train German-English NMT models on data selected by using the test set (source), the approximated target side, and a mixture of both. Our findings reveal that models built using a combination of outputs of FDA (using the test set and an approximated target side) perform better than those solely using the test set. We obtain a statistically significant improvement of more than 1.5 BLEU points over a model trained with all data, and more than 0.5 BLEU points over a strong FDA baseline that uses source-side information only. 1. Introduction Supervised machine learning aims to learn predictive models from a set of labeled examples (training data) so that it can accurately predict the labels of new, unlabeled, examples. Having more data may seem at first glance to be beneficial to building more accurate models, but upon closer inspec- tion this is not necessarily always the case. Machine learning models by design have an inductive bias that forces them to generalize over the training examples rather than just mem- orizing them without generalization. This means, however, that if the size of the training set is increased, this may lead to optimizing the model for predicting the labels of more ex- amples, but which on average are less relevant at test time than would be the case for a more focused, smaller train- ing set. The intuition of the importance of using a highly relevant set of training examples is captured well by the K- nearest neighbour model, which essentially computes at test time on-the-fly a very localized density estimate for every test example, based on the K training examples closest to the test example. It then uses this density estimate for classifi- cation. For the K-nearest neighbour model, increasing K too much is …

Content truncated to 500 words