OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora

OpenSubtitles2018: Statistical Rescoring of Sentence Alignments in Large, Noisy Parallel Corpora Pierre Lison∗, Jörg Tiedemann†, Milen Kouylekov‡ ∗Norwegian Computing Center plison@nr.no † Department of Modern Languages, University of Helsinki jorg.tiedemann@helsinki.fi ‡ Department of Informatics, University of Oslo milen.kouylekov@usit.uio.no Abstract Movie and TV subtitles are a highly valuable resource for the compilation of parallel corpora thanks to their availability in large numbers and across many languages. However, the quality of the resulting sentence alignments is often lower than for other parallel corpora. This paper presents a new major release of the OpenSubtitles collection of parallel corpora, which is extracted from a total of 3.7 million subtitles spread over 60 languages. In addition to a substantial increase in the corpus size (about 30 % compared to the previous version), this new release associates explicit quality scores to each sentence alignment. These scores are determined by a feedforward neural network based on simple language-independent features and estimated on a sample of aligned sentence pairs. Evaluation results show that the model is able predict lexical translation probabilities with a root mean square error of 0.07 (coefficient of determination R2 = 0.47). Based on the scores produced by this regression model, the parallel corpora can be filtered to prune out low-quality alignments. Keywords: Parallel Corpora, Machine Translation, Bitext alignment 1 Introduction Movie and TV subtitles are used in a wide range of lan- guage technology applications. Their availability in a large number of languages makes them well-suited for the cre- ation of parallel multilingual corpora. These corpora are a central resource for learning machine translation models (Koehn, 2009) but can also be used for corpus-driven lex- icography, cross-lingual NLP or translation research (Paet- zold, 2016; Akbik et al., 2016; Mikhailov and Cooper, 2016). Recent work on neural conversation models also showed that subtitles can be used to train dialogue agents (Vinyals and Le, 2015; Lison and Bibauw, 2017). Parallel corpora derived from subtitles have a number of benefits. The first one is their size: the OpenSubtitles dataset is (to the best of our knowledge) the world’s largest open collection of parallel corpora. The latest release, which is presented in this paper, contains no less than 3.4 billion sentences (amounting to 22.2 billion tokens) spread over 60 languages and a total of 1782 language pairs. As subtitles are annotated with timestamps, they can also be ef- ficiently synchronised using a linear-time algorithm (Tiede- mann, 2008). Finally, their conversational nature make them ideal for exploring dialogue phenomena and prop- erties of everyday language (Paetzold and Specia, 2016; van der Wees et al., 2016). However, the extraction of parallel corpora from subtitles must also face some challenges. One difficulty stems from the fact that subtitles are typicall not direct translations of one another. Rather, they should better be viewed as boiled- down transcriptions of the same conversations across sev- eral languages. Subtitles will inevitably differ in how they “compress” the conversations, notably due to structural di- vergences between languages, cultural differences and dis- parities in subtitling traditions/conventions. As …

Content truncated to 500 words