A Bilingual Discourse Corpus and Its Applications

A Bilingual Discourse Corpus and Its Applications Yang Liu1, Jiajun Zhang1, Chengqing Zong1, Yating Yang2 and Xi Zhou2 1NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China 2Xinjiang Technical Institute of Physics & Chemistry, Chinese Academy of Sciences, Urumqi, Xinjiang, China {yang.liu2013, jjzhang, cqzong}@nlpr.ia.ac.cn {yangyt,zhouxi}@ms.xjb.ac.cn Abstract Existing discourse research only focuses on the monolingual languages and the inconsistency between languages limits the power of the discourse theory in multilingual applications such as machine translation. To address this issue, we design and build a bilingual discource corpus in which we are currently defining and annotating the bilingual elementary discourse units (BEDUs). The BEDUs are then organized into hierarchical structures. Using this discourse style, we have annotated nearly 20K LDC sentences. Finally, we design a bilingual discourse based method for machine translation evaluation and show the effectiveness of our bilingual discourse annotations. Keywords: discourse analysis, bilingual discourse parser, SMT metric with discourse information 1. Introduction Discourse theory has been studied for decades. The task of discourse analysis is to segment sentences into non-overlap elementary discourse units (EDU) and then reorganize them via discourse relations to form discourse structures such as linear chains(Eisenstein and Barzilay, 2008), trees (Feng and Hirst, 2013) and graphs (Wolf and Gibson, 2005) in various discourse banks e.g. RST and PDTB. Due to the semantic integrity of EDUs, EDU relations and their well-formed structures, discourse knowledge has been applied to many natural language processing (NLP) tasks, such as information extraction, summarization, QA and statistical machine translation (SMT). Previous research works have proven that discourse information is beneficial to these NLP tasks. However, the current discourse annotations concentrating only on monolingual languages have some insufficiency that limits its power in many multilingual NLP tasks. Let us take statistical machine translation for example. Machine translation aims at finding for the source language sentence a target language sentence, which shares the same meaning as its source side. Intuitively, the source sentence and its target translation should have the similar discourse structure. Several recent research works apply the monolingual discourse knowledge to SMT, source side or target side. Some of them enforce the target language translation to be consistent with the source-side discourse structure (Tu et al., 2014). The others attempt to measure the translation discourse structure using discourse structures of target language references (Joty et al.,2014). Since there are wide variations in discourse annotations between source language and target language, the above two methods are somewhat conflicting. The problem is similar in other multilingual NLP tasks e.g. multilingual summarization (Anechitei and Ignat, 2013) ,discourse translation (Marcu et al., 2000; Tu et al., 2013) and bilingual semantic role labeling (Yang et al., 2015). Through the above analysis, it is obvious to realize that developing multilingual discourse corpus is quite necessary for multilingual NLP tasks and helpful to understand the diversity of discourse information in multilingual environment. Therefore, at the start point, we propose and annotate a bilingual discourse corpus. Furthermore, we design a MT evaluation metric based on the bilingual discourse annotations. …

Content truncated to 500 words