Automatic Construction of Discourse Corpora for Dialogue Translation

Automatic Construction of Discourse Corpora for Dialogue Translation Longyue Wang∗, Xiaojun Zhang∗, Zhaopeng Tu†, Andy Way∗, Qun Liu∗ ∗ ADAPT Centre, School of Computing, Dublin City University, Ireland † Noah Ark Lab, Huawei Technologies, China {lwang, xzhang, away, qliu}@computing.dcu.ie, tu.zhaopeng@huawei.com Abstract In this paper, a novel approach is proposed to automatically construct parallel discourse corpus for dialogue machine translation. Firstly, the parallel subtitle data and its corresponding monolingual movie script data are crawled and collected from Internet. Then tags such as speaker and discourse boundary from the script data are projected to its subtitle data via an information retrieval approach in order to map monolingual discourse to bilingual texts. We not only evaluate the mapping results, but also integrate speaker information into the translation. Experiments show our proposed method can achieve 81.79% and 98.64% accuracy on speaker and dialogue boundary annotation, and speaker-based language model adaptation can obtain around 0.5 BLEU points improvement in translation qualities. Finally, we publicly release around 100K parallel discourse data with manual speaker and dialogue boundary annotation. Keywords: Discourse Corpus, Dialogue, Machine Translation, Information Retrieval, Movie Script, Movie Subtitle 1. Introduction Dialogue is an essential component of social behaviour to express human emotions, moods, attitudes and personality. To date, few researchers have investigated how to improve the machine translation (MT) of conversational material by exploiting their internal structure. This lack of research on the dialogue MT is a surprising fact, since dialogue exhibits more cohesiveness than single sentence and at least as much than textual discourse. Although there are a number of papers on corpus construc- tion for various natural language processing (NLP) tasks, dialogue corpora are still scarce for MT. Some work re- garding bilingual subtitles as parallel corpora exists, but it lacks rich information between utterances (sentence-level corpus) (Lavecchia et al., 2007; Tiedemann, 2007a; Tiede- mann, 2007b; Itamar and Itai, 2008; Tiedemann, 2008; Xiao and Wang, 2009; Tiedemann, 2012; Zhang et al., 2014). Other work focuses on mining the internal struc- ture in dialogue data from movie scripts. However, these are monolingual data which cannot used for MT (Danescu- Niculescu-Mizil and Lee, 2011; Banchs, 2012; Walker et al., 2012; Schmitt et al., 2012). In general, the fact is that bilingual subtitles are ideal resources to extract paral- lel sentence-level utterances, and movie scripts contain rich information such as dialogue boundaries and speaker tags. Inspired by these facts, our initial idea was to build dia- logue discourse corpus by bridging the information in these two kinds of resources (i.e., scripts and subtitles). The cor- pus should be parallel, align at the segment-level as well as contain rich dialogue information. We propose a simple but effective approach to build our dialogue corpus. Firstly, we extract parallel sentences from bilingual subtitles, and mine dialogue information from monolingual movie scripts. Sec- ondly, we project dialogue information from script utter- ances to its corresponding parallel subtitle sentences us- ing an information retrieval (IR) approach. Finally, we ap- ply this approach to build a Chinese–English dialogue cor- pus, and …

Content truncated to 500 words