An exploratory approach to the Parallel Corpus Filtering shared task WMT20

Proceedings of the 5th Conference on Machine Translation (WMT), pages 959–965 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 959 An Exploratory Approach to the Corpus Filtering Shared Task WMT20 Ankur Kejriwal Department of Computer Science Johns Hopkins University akejriw2@jhu.edu Philipp Koehn Center for Language and Speech Processing Johns Hopkins University phi@jhu.edu Abstract This document describes an exploratory look into the Parallel Corpus Filtering Shared Task in WMT20. We submitted scores for both Pashto-English and Khmer-English systems combining multiple techniques like monolin- gual language model scores, length based fil- ters, language ID filters with confidence and norm of embedings. 1 Introduction For this task the participants were provided with a corpus of parallel data in Pashto-English (ps-en) and Khmer-English (km-en). Additional parallel and monolingual datasets were also provided. The task organizers built neural machine translation (NMT) systems from the scores produced, based on parallel training sets of 5 million words. These systems are sensitive towards noise (Khayrallah and Koehn, 2018) and thus, it becomes important to separate the useful data from the noise. We view the task as data that passes through a pipeline of filters in order to give us the best possible selection of 5 million words in the end. We determined that language ID filtering is a very strong pre-processing filter and inducting con- fidence scores is not needed. We also determined that monolingual Language models can help us in selecting sentences even if both the source and tar- get language models are independent of each other. Finally, using using the length of a sentence as a filter helps us create a better NMT system. We also learn that statistical intuitions do not easily extend to neural embeddings. 2 Baseline The idea behind our baseline system is to use the cosine distance between two multilingual represen- tations as a notion of parallelism between sentences embedded in the same space. The tool we use for this is LASER1 which uses an encoder-decoder architecture to train a multilingual sentence rep- resentation model. It has been shown by Artetxe and Schwenk (2018b) that LASER is effective at zero-shot cross-lingual natural language inference in the XNLI dataset which makes it promising for this task involving low-resource languages. Koehn et al. (2019) has shown this to be a strong baseline. We follow the work done by Artetxe and Schwenk (2018a) and begin by generating mul- tilingual sentence embeddings using LASER. The LASER score is a function of the margin between the cosine similarity between a given candidate and it’s k nearest neighbors2: Let f(x,y) = � zϵNNk(x) cos(x, z) 2k + � zϵNNk(y) cos(y, z) 2k LASER score(x,y) = margin(cos(x,y),f(x,y)) We experiment with the following definitions of margin: • Absolute: margin(a,b) = a Essentially just cosine similarity. • Distance: margin(a,b) = a - b Subtracting the average cosine similarity from that of the given candidate. We use this when there are certain points that are extra- ordinarily close to many other data points and thus, degrade the quality of nearest neighbors. …

Content truncated to 500 words