Parallel Corpus Filtering via Pre-trained Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8545–8554 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 8545 Parallel Corpus Filtering via Pre-trained Language Models Boliang Zhang, Ajay Nagesh, and Kevin Knight DiDi Labs {boliangzhang, ajaynagesh, kevinknight}@didiglobal.com Abstract Web-crawled data provides a good source of parallel corpora for training machine transla- tion models. It is automatically obtained, but extremely noisy, and recent work shows that neural machine translation systems are more sensitive to noise than traditional statistical ma- chine translation methods. In this paper, we propose a novel approach to filter out noisy sentence pairs from web-crawled corpora via pre-trained language models. We measure sen- tence parallelism by leveraging the multilin- gual capability of BERT and use the Genera- tive Pre-training (GPT) language model as a domain filter to balance data domains. We evaluate the proposed method on the WMT 2018 Parallel Corpus Filtering shared task, and on our own web-crawled Japanese-Chinese parallel corpus. Our method significantly out- performs baselines and achieves a new state- of-the-art. In an unsupervised setting, our method achieves comparable performance to the top-1 supervised method. We also evalu- ate on a web-crawled Japanese-Chinese paral- lel corpus that we make publicly available. 1 Introduction Training modern neural machine translation (NMT) systems requires large parallel-text resources. Publicly-available parallel corpora are mostly paired with English, such as German-English, French-English, Chinese-English, etc., and their domains are limited. For building machine transla- tion systems between non-English language pairs, such as Chinese and Japanese, existing parallel corpora are insufficient and often low quality. To address this problem, system builders have trained NMT systems on web-crawled data and achieved promising results (Xu and Koehn, 2017; Junczys- Dowmunt, 2018; Schwenk, 2018; Schwenk et al., 2019). However, data automatically crawled from the web is extremely noisy. Khayrallah and Koehn (2018) and Belinkov and Bisk (2018) show that neural translation models are far more sensitive to noisy parallel training data than statistical machine translation. Data selection methods that can fil- ter noisy parallel sentences from large-scale web crawled resources are in demand. In this paper, we study the problem in a real- world scenario where we crawl a large Japanese- Chinese parallel corpus from various websites and build open-domain machine translation systems between Japanese and Chinese, by filtering the web crawled parallel corpus. In addition, a small amount of clean parallel data is available, in the software domain. In order to confirm our results on a public data, we also apply our filter to the WMT 2018 German-English Parallel Corpus Fil- tering shared task. Previous work on parallel corpus filtering per- forms poorly in our scenario as it either requires large clean parallel corpora or dictionaries (Xu and Koehn, 2017; Artetxe and Schwenk, 2019; Junczys-Dowmunt, 2018; Chaudhary et al., 2019), or relies on multilingual word embeddings and ne- glects context when measuring translation paral- lelism (Hangya and Fraser, 2018). In this paper, we propose a simple but effec- tive parallel corpus filtering method. Multilingual BERT (Devlin et al., …

Content truncated to 500 words