Filtering Noisy Parallel Corpus using Transformers with Proxy Task Learning

Proceedings of the 5th Conference on Machine Translation (WMT), pages 940–946 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 940 Filtering Noisy Parallel Corpus using Transformers with Proxy Task Learning Haluk Acarcicek1, Talha Colakoglu1, Pinar Ece Aktan1, Chongxuan Huang2, Wei Peng2 ∗ 1Turkey R&D AI Enablement Department, Huawei Technologies {haluk.acarcicek,talha.colakoglu,ece.aktan.hatipoglu}@huawei.com 2Artificial Intelligence Application Research Center, Huawei Technologies {huang.chongxuan,peng.wei1}@huawei.com Abstract This paper illustrates Huawei’s submission to the WMT20 low-resource parallel corpus fil- tering shared task. Our approach focuses on developing a proxy task learner on top of a transformer-based multilingual pre-trained lan- guage model to boost the filtering capability for noisy parallel corpora. Such a supervised task also helps us to iterate much more quickly than using an existing neural machine transla- tion system to perform the same task. After performing empirical analyses of the finetun- ing task, we benchmark our approach by com- paring the results with past years’ state-of-the- art records. This paper wraps up with a dis- cussion of limitations and future work. The scripts for this study will be made publicly available. 1 1 Introduction Crawling web has been regarded as a de facto ap- proach to produce bitexts, yet the crawled texts are under-qualified often in some aspects to train a proper machine translation system. Under- qualified bitexts present misalignments, no align- ments, wrong language pairs, sentences mostly composed of numbers and mathematical formulas, etc. Parallel corpus filtering in this manner holds a critical research area to improve the performance of machine translation systems. WMT organizes a shared task for parallel corpus filtering since 2018 intending to filter our noisy bitexts to this end. The challenge targets low-resource language pairs since 2019. Many existing filtering methods require multiple layers of elimination by implementing manually en- gineered features such as length filtering, language identification, normalizing, etc. These hand-picked features work well for a language pair but don’t ∗Corresponding author 1https://github.com/WPti/proxy-filter generalize well to another language pair or domain and often bring algorithmic complexity to the over- all system. The LASER (Artetxe and Schwenk, 2019) model achieved state-of-the-art (SOTA) records at the WMT19 shared task on low-resource parallel cor- pus filtering (Chaudhary et al., 2019). The sentence representation model implemented in LASER pro- vides a means for measuring the similarity between a source and a target sentence. As stated in the future work at Artetxe and Schwenk (2019), there is still space to improve. Utilizing a self-attention mechanism remains future work as the LASER was not built upon the latest transformer architecture (Vaswani et al., 2017). We are also interested in designing a filtering tool that can be efficiently ap- plied to a wide range of language pairs. Pre-trained multilingual language models, such as BERT (De- vlin et al., 2018) and RoBERTa (Liu et al., 2019), are exploited to this end. We make two contributions to the field in this manner. The first contribution is a proposal of approaching the filtering problem as a discrimi- nation task that can be trained with a proxy task and synthetic …

Content truncated to 500 words