JESC: Japanese-English Subtitle Corpus

JESC: Japanese-English Subtitle Corpus Reid Pryzant1, Youngjoo Chung2, Dan Jurafsky1, Denny Britz3 1Stanford University, 2Rakuten Institute of Technology, 3Google Brain {rpryzant, jurafsky}@stanford.edu, yjchung@acm.org, dennybritz@gmail.com Abstract In this paper we describe the Japanese-English Subtitle Corpus (JESC). JESC is a large Japanese-English parallel corpus covering the underrepresented domain of conversational dialogue. It consists of more than 3.2 million examples, making it the largest freely available dataset of its kind. The corpus was assembled by crawling and aligning subtitles found on the web. The assembly process incorporates a number of novel preprocessing elements to ensure high monolingual fluency and accurate bilingual alignments. We summarize its contents and evaluate its quality using human experts and baseline machine translation (MT) systems. Keywords: parallel corpus, asian languages, machine translation 1. Introduction There is a strong need for large parallel corpora from new domains. Modern machine translation (MT) systems are fundamentally constrained by the availability and quantity of parallel corpora. Apart from the exceptions of English- Arabic, English-Chinese, and several European pairs, par- allel corpora remain a scarce resource due to the high cost of manual construction (Chu et al., 2014). Furthermore, despite promising work in domain adaptation, MT systems struggle to generalize to new domains that are disparate from their training data (Pryzant et al., 2017). This need for large, novel-domain data is especially evi- dent in the resource-poor Japanese-English (JA-EN) lan- guage pair. Only two large (>1M phrase pairs) and free datasets exist for this language pair (Neubig, 2017; Tiede- mann, 2017; Moses, 2017). The first is called ASPEC. It consists of 3M examples and it originates from scien- tific papers, a highly formalized and written domain (all other JA-EN datasets have similar language) (Nakazawa et al., 2016). The other, OpenSubtitles, is a multi-language dataset of aligned subtitles authored by professional trans- lators; the JA-EN subset of these data contains approxi- mately 1M examples (Lison and Tiedemann, 2016). Open- Subtitles is to the best of these authors knowledge the only parallel corpus to cover the unrepresented domains of con- versational speech and informal writing. This dearth of large-scale and informal data is especially problematic be- cause colloquial Japanese has significant structural charac- teristics which can preclude cross-domain translation (Tsu- jimura, 2013). We hope to alleviate this problem by build- ing off the work of (Lison and Tiedemann, 2016) to con- struct a larger corpus that incorporates the vast number of unofficial and fan-made subtitles on the web. Subtitles are an excellent source for alleviating resource scarcity problems. There are a wide and interesting range of linguistic phenomena in subtitles that are poorly repre- sented elsewhere. This includes colloquial exchange, slang, expository discourse, dialect, vernacular, and movie dialog, which is available in great quantities and has been shown to resemble natural conversation (Forchini, 2013). Further- more, large subtitle databases are freely available on the web, are often crowd-sourced, and the close correspon- dence between subtitles and their video material renders time-based alignment feasible (Tiedemann, 2008). We release JESC, a new Japanese-English parallel corpus consisting of 3.2 …

Content truncated to 500 words