Detecting Various Types of Noise for Neural Machine Translation

Findings of the Association for Computational Linguistics: ACL 2022, pages 2542 - 2551 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Detecting Various Types of Noise for Neural Machine Translation Christian Herold Jan Rosendahl Joris Vanvinckenroye Human Language Technology and Pattern Recognition GroupHermann Ney Computer Science Department RWTH Aachen University D-52056 Aachen, Germany {surname}@i6.informatik.rwth-aachen.de Abstract The filtering and/or selection of training data is one of the core aspects to be considered when building a strong machine translation sys- tem. In their influential work, Khayrallah and Koehn (2018) investigated the impact of dif- ferent types of noise on the performance of machine translation systems. In the same year the WMT introduced a shared task on parallel corpus filtering, which went on to be repeated in the following years, and resulted in many dif- ferent filtering approaches being proposed. In this work we aim to combine the recent achieve- ments in data filtering with the original analysis of Khayrallah and Koehn (2018) and investi- gate whether state-of-the-art filtering systems are capable of removing all the suggested noise types. We observe that most of these types of noise can be detected with an accuracy of over 90% by modern filtering systems when operating in a well studied high resource set- ting. However, we also find that when con- fronted with more refined noise categories or when working with a less common language pair, the performance of the filtering systems is far from optimal, showing that there is still room for improvement in this area of research. 1 Introduction The phenomenon of noisy data in the training of machine translation (MT) systems has been studied from various angles over recent years. To outline the impact of noise, Khayrallah and Koehn (2018) specified ten common noise categories and syn- thetically generated noisy data samples for each of them. By adding the noisy samples to an otherwise clean training corpus they measured the effect on the resulting translation system. Their conclusion was that neural machine translation (NMT) is less robust towards noisy data than statistical machine translation and that some noise types can prove very detrimental to NMT performance. As NMT had surpassed the statistical approaches just a few years prior, this work paved the way for a spiked interest in data filtering research for machine trans- lation. In the same year, the Conference on Machine Translation (WMT) started to host an annual shared task on parallel corpus filtering (Koehn et al., 2018, 2019, 2020) featuring a broad mix of academic and industrial submissions. This shared task highlights the general need for well working data filtering sys- tems and resulted in the publication of a variety of new filtering approaches (Junczys-Dowmunt, 2018; Chaudhary et al., 2019; Lu et al., 2020). The WMT evaluations simulate a real-world data filtering task on web crawled data. Each participating system is required to extract a fixed amount of parallel data and is ranked according to the performance of the translation system that was trained on the selected data. While this …

Content truncated to 500 words