OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles

OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles Pierre Lison∗, J¨org Tiedemann† ∗ Department of Informatics University of Oslo plison@ifi.uio.no † Department of Modern Languages University of Helsinki jorg.tiedemann@helsinki.fi Abstract We present a new major release of the OpenSubtitles collection of parallel corpora. The release is compiled from a large database of movie and TV subtitles and includes a total of 1689 bitexts spanning 2.6 billion sentences across 60 languages. The release also incorporates a number of enhancements in the preprocessing and alignment of the subtitles, such as the automatic correction of OCR errors and the use of meta-data to estimate the quality of each subtitle and score subtitle pairs. Keywords: Parallel corpora, Bitext alignment, Statistical Machine Translation 1 Introduction Movie and TV subtitles constitute a prime resource for the compilation of parallel corpora. From a linguistic per- spective, subtitles cover a wide and interesting breadth of genres, from colloquial language or slang to narrative and expository discourse (as in e.g. documentaries). Large databases of subtitles are also available and continue to grow rapidly – for instance, the OpenSubtitles1 database contains more than 3 million subtitles in over 60 languages. Finally, and perhaps most importantly, the tight connection between subtitles and their corresponding source material – usually a movie or TV episode – makes it possible to efficiently align subtitles across languages based on time overlaps (Tiedemann, 2007). This paper presents a new release of the OpenSubtitles col- lection of parallel corpora. The new release includes a total of 1689 bitexts extracted from a collection of subtitles of 2.6 billion sentences (17.2 billion tokens) distributed over 60 languages. In addition to increasing the global volume of the dataset by approximately 39 % compared to the pre- vious version (Tiedemann, 2012), the release also includes several important improvements regarding how the subti- tles are preprocessed and aligned with one another. Figure 1 illustrates the processing workflow, starting with the raw subtitle files and ending with the resulting bitexts. Sections 2 and 3 present the dataset and the preprocessing techniques employed to convert the subtitle files into sen- tences and tokens encoded in XML. Section 4 describes the alignment process for cross-lingual alignments, and Section 5 extends it to alignments of alternative subtitles within the same language. Section 6 concludes this paper. 2 Source Data The dataset consists of a database dump of the OpenSub- titles.org repository of subtitles, comprising a total of 3.36 million subtitle files covering more than 60 languages. We filtered out the languages associated with less than 10 sub- 1http://www.opensubtitles.org. titles, as they are typically the result of human classifica- tion errors. Most files are encoded in the .srt format. Subtitles that are encoded in an alternative format are first converted to .srt before processing. The following infor- mation is provided for each subtitle: 1. A unique identifier, 2. A list of files (there may be more than one file in the case of movies with multiple CDs), 3. A language code and subtitle format, 4. …

Content truncated to 500 words