A Corpus of Word-Aligned Asked and Anticipated Questions in a Virtual Patient Dialogue System

A Corpus of Word-Aligned Asked and Anticipated Questions in a Virtual Patient Dialogue System Ajda Gokcen, Evan Jaffe, Johnsey Erdmann, Michael White, Douglas Danforth The Ohio State University Department of Linguistics, Department of Family Medicine {gokcen.2,jaffe.59,erdmann.10}@osu.edu, mwhite@ling.osu.edu doug.danforth@osumc.edu Abstract We present a corpus of virtual patient dialogues to which we have added manually annotated gold standard word alignments. Since each question asked by a medical student in the dialogues is mapped to a canonical, anticipated version of the question, the corpus implicitly defines a large set of paraphrase (and non-paraphrase) pairs. We also present a novel process for selecting the most useful data to annotate with word alignments and for ensuring consistent paraphrase status decisions. In support of this process, we have enhanced the earlier Edinburgh alignment tool (Cohn et al., 2008) and revised and extended the Edinburgh guidelines, in particular adding guidance intended to ensure that the word alignments are consistent with the overall paraphrase status decision. The finished corpus and the enhanced alignment tool are made freely available. Keywords: monolingual word alignment, paraphrase detection, virtual patient dialogues 1. Introduction Many systems for detecting paraphrases or measuring se- mantic similarity rely on alignments of words and phrases (whether explicitly or implicitly). However, to our knowl- edge only two monolingual corpora with manually anno- tated gold standard alignments have been developed to date, and none in a naturalistic task setting: the MSRP (Brockett, 2007) and Edinburgh (Cohn et al., 2008) corpora.1 More- over, both of these corpora have additional shortcomings. In particular, the Edinburgh corpus consists only of aligned paraphrase pairs, rather than containing both paraphrase and non-paraphrase pairs; and although the MSRP corpus does include a mix of paraphrase and non-paraphrase pairs, the word alignments were annotated without taking into consideration the paraphrase status of the sentence pair. This potentially makes the MSRP alignments less useful than they might otherwise be, given that Xu et al. (2014) have recently shown that there can be considerable benefit to modeling word alignment and paraphrase classification as a joint process. In this paper, we present a corpus of 104 dialogues between early stage medical students and a virtual standardized pa- tient (Figure 1) from which we have manually annotated word alignments for 942 sentence pairs. With this dialogue system (Danforth et al., 2013), the interpretation task is to determine for each question asked by the medical student which of the set of questions anticipated by the content au- thor best matches the student’s question. In previous work on developing a log-linear ranking model for this system (Jaffe et al., 2015), we found the interpretation task to be 1See Xu et al. (2014) for a review of paraphrase corpora without gold standard alignments. Also closely related is the much larger WikiAnswers Paraphrase Dataset (Fader et al., 2013), which again does not have manually annotated alignments and which exhibits much less topical coherence. quite challenging, since it requires discriminating among many topically related questions; by contrast, the MSRP task (Dolan et al., 2004) is …

Content truncated to 500 words