Dual Conditional Cross Entropy Scores and LASER Similarity Scores for the WMT20 Parallel Corpus Filtering Shared Task

Proceedings of the 5th Conference on Machine Translation (WMT), pages 966–971 Online, November 19–20, 2020. c⃝2020 Association for Computational Linguistics 966 Dual Conditional Cross Entropy Scores and LASER Similarity Scores for the WMT20 Parallel Corpus Filtering Shared Task Felicia Koerner Philipp Koehn Center for Language and Speech Processing Johns Hopkins University {fkr, phi}@jhu.edu Abstract This paper describes our submission to the WMT20 Parallel Corpus Filtering and Align- ment for Low-Resource Conditions Shared Task. This year’s corpora are noisy Khmer- English and Pashto-English, with 58.3 mil- lion and 11.6 million words respectively (En- glish token count). Our submission focuses on filtering Pashto-English, building on pre- viously successful methods to produce two sets of scores: LASER LM, a combination of the LASER similarity scores provided in the shared task and perplexity scores from lan- guage models, and DCCEF DUP, dual condi- tional cross entropy scores combined with a duplication penalty. We improve slightly on the LASER similarity score and find that the provided clean data can successfully be sup- plemented with a subsampled set of the noisy data, effectively increasing the training data for the models used for dual conditional cross entropy scoring. 1 Introduction Machine translation systems require large amounts of high quality parallel corpora for training. Neu- ral machine translation models in particular have been found to both require more data (Koehn and Knowles, 2017), and be more sensitive to noise in training data (Khayrallah and Koehn, 2018) than statistical machine translation models. While these data can be acquired from online sources, the re- sulting crawled texts are often noisy and require filtering to produce large amounts of sufficiently clean training data. 2 Related Work We refer readers to (Koehn et al., 2019) for a more detailed overview of methods for parallel corpus fil- tering, here we describe the most relevant methods to this work. 2.1 Rule-based Filtering Most filtering methods employ some rule-based fil- tering, usually to prepare the data for other scoring methods, based on language models, classifiers, or other translation models. (S´anchez-Cartagena et al., 2018) apply hard rules to filter out data before using a classifier to score sentence pairs. (Rossenbach et al., 2018) use many rules, including limits on sentence length, Levenshtein distance, length ratio, and token ratio. We use basic language ID and overlap rules only for the Dual Conditional Cross Entropy Scores, this is described in more detail in subsection 5.1. The LASER similarity scores provided by the shared task organizers also apply a language ID filter (assigning the pair a score of 0 if either of the sentences are not recognized as the expected language). 2.2 Dual Conditional Cross Entropy Scores The most successful scoring method in the WMT18 Shared Task on Parallel Corpus Filtering was Dual Conditional Cross Entropy Filtering (dccef) (Junczys-Dowmunt, 2018). This method trains an NMT model in both translation directions, uses these to calculate the cross-entropy for each sen- tence, and finally produces a score based on their agreement. As this year’s task deals with low- resource languages (contrary …

Content truncated to 500 words