Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection

Proceedings of the Third Conference on Machine Translation (WMT), Volume 1: Research Papers, pages 133–143 Belgium, Brussels, October 31 - Novermber 1, 2018. c⃝2018 Association for Computational Linguistics https://doi.org/10.18653/v1/W18-64014 Denoising Neural Machine Translation Training with Trusted Data and Online Data Selection Wei Wang Google Research wangwe@google.com Taro Watanabe Google Research tarow@google.com Macduff Hughes Google Research macduff@google.com Tetsuji Nakagawa Google Research tnaka@google.com Ciprian Chelba Google Research ciprianchelba@google.com Abstract Measuring domain relevance of data and iden- tifying or selecting well-fit domain data for machine translation (MT) is a well-studied topic, but denoising is not yet. Denoising is concerned with a different type of data qual- ity and tries to reduce the negative impact of data noise on MT training, in particular, neu- ral MT (NMT) training. This paper general- izes methods for measuring and selecting data for domain MT and applies them to denoising NMT training. The proposed approach uses trusted data and a denoising curriculum real- ized by online data selection. Intrinsic and extrinsic evaluations of the approach show its significant effectiveness for NMT to train on data with severe noise. 1 Introduction Data noise is an understudied topic in the machine translation (MT) field. Recent research has found that data noise has a bigger impact on neural ma- chine translation (NMT) than on statistical ma- chine translation (Khayrallah and Koehn, 2018), but learning what data quality (or noise) means in NMT and how to make NMT training robust to data noise remains an open research question. On the other hand, a rich body of MT data re- search focuses on domain data relevance and se- lection for domain adaptation purpose. As a re- sult, effective and successful methods have been published and shown to work for both SMT and NMT. For example, (Axelrod et al., 2011) intro- duce a metric for measuring the data relevance to a domain by using n-gram language models (LM). (van der Wees et al., 2017) employ a neural- network version of it and propose a gradually- refining strategy to dynamically schedule data dur- ing NMT training. In these methods, a large amount of in-domain data are used to help mea- sure data domain relevance. Data noise is a different quality that has been shown to affect NMT performance in particular. In MT, the use of web crawl, automatic methods for parallel data mining, sentence alignment pro- vide us with parallel data of variable quality from many points of view: sentence breaking, poor sen- tence alignments, translations, domain adequacy, tokenization and so forth. To deal with such data noise, a commonly used practice is (static) data filtering with simple heuristics or classifica- tion. The NMT community increasingly realizes that this type of quality matters for general NMT translation accuracy. For example, (Khayrallah and Koehn, 2018) studies the types of data noise and their impact on NMT; WMT 2018 introduces a Parallel Corpus Filtering task on noisy web- crawled data. Unfortunately, the ingredients that made do- main data selection methods successful have not been studied in the …

Content truncated to 500 words