Bifixer and Bicleaner: two open-source tools to clean your parallel data

Bifixer and Bicleaner: two open-source tools to clean your parallel data Gema Ram´ırez-S´anchez, Jaume Zaragoza-Bernabeu, Marta Ba˜n´on, Sergio Ortiz Rojas Prompsit Language Engineering, S.L. Campus UMH, Edifici Quorum III Av. de la Universitat, s/n. 03202. Elx. Spain {gramirez,jzaragoza,mbanon,sortiz}@prompsit.com Abstract This paper shows the utility of two open- source tools designed for parallel data cleaning: Bifixer and Bicleaner. Already used to clean highly noisy parallel con- tent from crawled multilingual websites, we evaluate their performance in a dif- ferent scenario: cleaning publicly avail- able corpora commonly used to train ma- chine translation systems. We choose four English–Portuguese corpora which we plan to use internally to compute para- phrases at a later stage. We clean the four corpora using both tools, which are de- scribed in detail, and analyse the effect of some of the cleaning steps on them. We then compare machine translation training times and quality before and after cleaning these corpora, showing a positive impact particularly for the noisiest ones. 1 Introduction Parallel corpora are usually the main source of in- formation used to learn machine translation mod- els. The availability of corpora has encouraged the advance of machine translation in both academy and industry settings. Publicly available parallel corpora (Europarl, News Commentary, United Na- tions, etc.) have been used for decades now, not only to produce machine translation but also other by-products such as dictionaries, concordances, synonyms, paraphrases, etc. In machine transla- tion, due to the ability of statistical models to hide imperfections without statistical significance, fil- tering out noise from these corpora was not very c⃝ 2020 The authors. This article is licensed under a Creative Commons 3.0 licence, no derivative works, attribution, CC- BY-ND. important. Now that neural models have super- seded statistical ones, we need to be more careful about noise in the input as it has a higher impact on the output, as discussed in (Khayrallah and Koehn, 2018) and (Rikters, 2018). Inspired by recent work on filtering parallel cor- pora to maximize the quality of machine transla- tion from the shared tasks organised at WMT181 and WMT192, we review how noisy some of the most popular or recent publicly available corpora are and how this impacts the quality of the output of state-of-the-art neural machine translation. Our motivation is twofold: getting high-quality mono- lingual and bilingual data and getting high-quality machine translation for English–Portuguese. We will further use this resources to compute para- phrases in the framework of a research project. In order to inspect and filter out noise, we use Bifixer and Bicleaner,3 a couple of publicly available cleaning tools released as part of the ParaCrawl European project.4 These tools have been mainly used to filter out noise from the raw version of automatically crawled parallel corpora in more than 30 language combinations. Here we use them in a very different scenario: we take al- ready released publicly available corpora, either widely used in the past or recent. We analyse the main problems of the corpora and review the clean- ing steps …

Content truncated to 500 words