Bicleaner at WMT 2020: Universitat d’Alacant-Prompsit’s submission to the parallel corpus filtering shared task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 952–958 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 952 Bicleaner at WMT 2020: Universitat d’Alacant–Prompsit’s submission to the parallel corpus filtering shared task Miquel Espl`a-Gomis⋆ V´ıctor M. S´anchez-Cartagena⋆ Jaume Zaragoza-Bernabeu† Felipe S´anchez-Mart´ınez⋆ ⋆Dep. de Llenguatges i Sistemes Inform`atics, Universitat d’Alacant E-03690 Sant Vicent del Raspeig (Spain) {mespla,vmsanchez,fsanchez}@dlsi.ua.es †Prompsit Language Engineering Av. Universitat s/n. Edifici Quorum III, E-03202 Elx (Spain) jzaragoza@prompsit.com Abstract This paper describes the joint submission of Universitat d’Alacant and Prompsit Language Engineering to the WMT 2020 shared task on parallel corpus filtering. Our submission, based on the free/open-source tool Bicleaner, enhances it with Extremely Randomised Trees and lexical similarity features that account for the frequency of the words in the parallel sen- tences to determine if two sentences are paral- lel. To train this classifier we used the clean corpora provided for the task and synthetic noisy parallel sentences. In addition we re- score the output of Bicleaner using character- level language models and n-gram saturation. 1 Introduction This paper describes the joint submission of Univer- sitat d’Alacant and Prompsit Language Engineer- ing to the parallel corpus filtering shared task at the Fifth Conference on Machine Translation (WMT 2020). Our submission is built upon Bicleaner (S´anchez-Cartagena et al., 2018),1 a widely-used free/open-source tool for detecting noisy parallel sentences that participated in the 2018 edition of this shared task and ranked fourth out of 17 submis- sions on one of the sub-tasks. We provide quality scores for the sentence pairs provided by the organ- iser without re-aligning them. The 2020 edition of the parallel corpus filter- ing shared task focuses on two under-resourced Asian languages paired with English: Khmer and Pashto. Khmer (km) is the official language of Cambodia and is spoken circa 16 million people in Cambodia, Vietnam and Thailand.2 There are about 500k English–Khmer parallel sentences in OPUS,3 mainly belonging to narrow domains like 1https://github.com/bitextor/bicleaner 2Wikipedia: https://en.wikipedia.org/wiki/ Khmer_language 3http://opus.nlpl.eu software products and religion. Pashto (ps) is spo- ken by around 40 million people in Pakistan and in Afghanistan, where it is official together with Persian.4 There are around 100k English–Pastho parallel sentence in OPUS, most of which belong to the software domain. Detecting noisy parallel sentences for under- resourced language pairs, like those addressed in this shared task, is challenging. Pastho is not di- rectly supported by LASER (Schwenk and Douze, 2017), although it supports other Iranian languages, and there are few bilingual resources for building the Bicleaner’s models. Bicleaner is based on a classifier that assesses whether a pair of sentences are mutual translations or not. It is trained on a parallel corpus (positive samples) and on an automatically corrupted ver- sion of the same corpus (negative samples). The most important features used by the classifier are lexical similarity scores obtained with the help of probabilistic bilingual dictionaries, which are also extracted from the parallel corpus. Our submis- sion improves the performance of the version of Bicleaner that took part in the 2018 shared …

Content truncated to 500 words