Norm-based Noisy Corpora Filtering and Refurbishing in Neural Machine Translation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5414 - 5425 December 7-11, 2022 ©2022 Association for Computational Linguistics Norm-based Noisy Corpora Filtering and Refurbishing in Neural Machine Translation Yu Lu1,2 and Jiajun Zhang1,2∗ 1 National Laboratory of Pattern Recognition, Institute of Automation, CAS, Beijing, China 2 School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China {yu.lu, jjzhang}@nlpr.ia.ac.cn Abstract Recent advances in neural machine translation depend on massive parallel corpora, which are collected from any open source without much guarantee of quality. It stresses the need for noisy corpora filtering, but existing methods are insufficient to solve this issue. They spend much time ensembling multiple scorers trained on clean bitexts, unavailable for low-resource languages in practice. In this paper, we pro- pose a norm-based noisy corpora filtering and refurbishing method with no external data and costly scorers. The noisy and clean samples are separated based on how much information from the source and target sides the model requires to fit the given translation. For the unparal- lel sentence, the target-side history translation is much more important than the source con- text, contrary to the parallel ones. The amount of these two information flows can be mea- sured by norms of source-/target-side context vectors. Moreover, we propose to reuse the discovered noisy data by generating pseudo labels via online knowledge distillation. Exten- sive experiments show that our proposed filter- ing method performs comparably with state-of- the-art noisy corpora filtering techniques but is more efficient and easier to operate. Noisy sam- ple refurbishing further enhances the perfor- mance by making the most of the given data1. 1 Introduction Neural machine translation (NMT) has achieved significant progress with help from large parallel corpora for training (Tiedemann, 2012; Smith et al., 2013). These data are typically extracted from the web without much control over the quality, which presents misalignment, wrong languages, too many numbers or URLs, etc. In this case, noisy corpora filtering holds a critical research area to prevent noisy bitexts from degrading the generalization per- formance of NMT (Khayrallah and Koehn, 2018). ∗Corresponding author. 1https://github.com/yulu-dada/Norm_NoisyFiltering wang yong@@ zhi [ 37@@ 69 30@@ 57 18@@ 07 ] is the chief designer in china ’s manned spaceflight project . Src: Tgt: Source-side Our metric Target-side [ 37@@ 69 30@@ 57 18@@ 07 ] in China Figure 1: An example of the required source-/target-side information when the model fits an unparallel Zh⇒En sentence pair (words in red are real noisy segments). The amount of information on each side is counted by norms of corresponding context vectors, positively correlated to the darkness of color blocks. Much effort has been devoted to this field with the promotion of a WMT shared task for parallel corpus filtering. However, prior work is difficult to operate in practice due to two drawbacks. (1) High time and computational cost. Their good perfor- mance relies on the ensembling of multiple large- scale scorers, which involves costly pre-training and fine-tuning (Esplà-Gomis et al., 2020; Lu …

Content truncated to 500 words