Analyzing the Use of Influence Functions for Instance-Specific Data Filtering in Neural Machine Translation

Proceedings of the Seventh Conference on Machine Translation (WMT), pages 295 - 309 December 7-8, 2022 ©2022 Association for Computational Linguistics Analyzing the Use of Influence Functions for Instance-Specific Data Filtering in Neural Machine Translation Tsz Kin Lam∗ ICL, Heidelberg University lam@cl.uni-heidelberg.de Eva Hasler Amazon AI Translate ehasler@amazon.com Felix Hieber Amazon AI Translate fhieber@amazon.com Abstract Customer feedback can be an important signal for improving commercial machine translation systems. One solution for fixing specific trans- lation errors is to remove the related erroneous training instances followed by re-training of the machine translation system, which we refer to as instance-specific data filtering. Influence functions (IF) have been shown to be effec- tive in finding such relevant training examples for classification tasks such as image classifi- cation, toxic speech detection and entailment task. Given a probing instance, IF find influen- tial training examples by measuring the simi- larity of the probing instance with a set of train- ing examples in gradient space. In this work, we examine the use of influence functions for Neural Machine Translation (NMT). We pro- pose two effective extensions to a state of the art influence function and demonstrate on the sub-problem of copied training examples that IF can be applied more generally than hand- crafted regular expressions. 1 Introduction Neural Machine Translation (NMT) is the de facto standard for recent high-quality machine transla- tion systems. NMT, however, requires abundant amount of bi-text for supervised training. One com- mon approach to increase the amount of bi-text is via data augmentation (Sennrich et al., 2015; Edunov et al., 2018; He et al., 2019, inter alia). Another approach is the use of web-crawled data (Bañón et al., 2020) but since crawled data is known to be notoriously noisy (Khayrallah and Koehn, 2018; Caswell et al., 2020), a plethora of data filtering techniques (Junczys-Dowmunt, 2018; Wang et al., 2018; Ramírez-Sánchez et al., 2020, in- ter alia) have been proposed for retaining a cleaner portion of the bi-text for training. While standard data filtering techniques aim to improve the quality of the overall training data ∗ Work done during an internship at Amazon. without targeting the translation quality of specific instances, instance-specific data filtering focuses on the improvement of translation quality toward a specific set of input sentences via removal of the related training data. In commercial MT, this selected set of sentences can be the problematic translations reported by customers. One simple approach of instance-specific data filtering in NMT is manual filtering. In manual filtering, human annotators identify translation errors on sentences reported by customer and designs filtering scheme, e.g., regular expressions to search related training examples for removal from the training set. In this work, we attempt to apply a more au- tomatable technique called influence functions (IF) which is shown to be effective on image classifi- cation (Koh and Liang, 2017), and certain NLP tasks such as sentiment analysis, entailment and toxic speech detection (Han et al., 2020; Guo et al., 2020). Given a probing example, influence func- tions (IF) …

Content truncated to 500 words