Dynamically Composing Domain-Data Selection with Clean-Data Selection by “Co-Curricular Learning” for Neural Machine Translation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1282–1292 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1282 Dynamically Composing Domain-Data Selection with Clean-Data Selection by “Co-Curricular Learning” for Neural Machine Translation Wei Wang and Isaac Caswell and Ciprian Chelba Google Research {wangwe,icaswell,ciprianchelba}@google.com Abstract Noise and domain are important aspects of data quality for neural machine translation. Existing research focus separately on domain- data selection, clean-data selection, or their static combination, leaving the dynamic in- teraction across them not explicitly examined. This paper introduces a “co-curricular learn- ing” method to compose dynamic domain-data selection with dynamic clean-data selection, for transfer learning across both capabilities. We apply an EM-style optimization procedure to further refine the “co-curriculum”. Exper- iment results and analysis with two domains demonstrate the effectiveness of the method and the properties of data scheduled by the co- curriculum. 1 Introduction Significant advancement has been witnessed in neural machine translation (NMT), thanks to bet- ter modeling and data. As a result, NMT has found successful use cases in, for example, domain trans- lation and helping other NLP applications, e.g., (Buck et al., 2018; McCann et al., 2017). As these tasks start to scale to more domains, a challenge starts to surface: Given a source monolingual cor- pus, how to use it to improve an NMT model to translate same-domain sentences well? Data se- lection plays an important role in this context. In machine translation, data selection has been a fundamental research topic. One idea (van der Wees et al., 2017; Axelrod et al., 2011) for this problem is to use language models to select par- allel data out of a background parallel corpus, seeded by the source monolingual sentences. This approach, however, performs poorly on noisy data, such as large-scale, web-crawled datasets, because data noise hurts NMT performance (Khayrallah and Koehn, 2018). The lower learning curve in Figure 1: BLEU curves over NMT training steps: domain- data selection on Paracrawl English→French data (lower curve) vs. clean-data selection on the same data (upper curve). Setup available in the experiment section. Figure 1 shows the effect of noise on domain-data selection. NMT community has realized the harm of data noise to translation quality, leading to efforts in data denoising (Koehn et al., 2018), as has been popular in computer vision (Hendrycks et al., 2018). The upper curve in Figure 1 shows the ef- fect of clean-data selection on the same noisy data. These denoising methods, however, cannot be di- rectly used for the problem in question as they re- quire trusted parallel data as input. We introduce a method to dynamically com- bine clean-data selection and domain-data selec- tion. We treat them as independent curricula, and compose them into a “co-curriculum”. We sum- marize our contributions as: 1. “Co-curricular learning”, for transfer learn- ing across data quality. It extends the single curriculum learning work in NMT and makes the existing domain-data selection method work better with noisy data. 2. A curriculum optimization …

Content truncated to 500 words