Estimation of Discourse Segmentation Labels from Crowd Data

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2190–2200, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Estimation of Discourse Segmentation Labels from Crowd Data Ziheng Huang Department of Statistics Columbia University zh2220@columbia.edu Jialu Zhong Department of Computer Science Columbia University jialu.zhong@columbia.edu Rebecca J. Passonneau Center for Computational Learning Systems Columbia University becky@ccls.columbia.edu Abstract For annotation tasks involving independent judgments, probabilistic models have been used to infer ground truth labels from data where a crowd of many annotators labels the same items. Such models have been shown to produce results superior to taking the majority vote, but have not been applied to sequential data. We present two methods to infer ground truth labels from sequential annotations where we assume judgments are not independent, based on the observation that an annotator’s segments all tend to be several utterances long. The data consists of crowd labels for anno- tation of discourse segment boundaries. The new methods extend Hidden Markov Models to relax the independence assumption. The two methods are distinct, so positive labels proposed by both are taken to be ground truth. In addition, results of the models are checked using metrics that test whether an annotator’s accuracy relative to a given model remains consistent across different conversations. 1 Introduction A single, spontaneous, spoken interaction can consist of multiple activities, such as to plan a future event, to complain about a past situation, or to carry out a transaction that might consist of subtasks. Speakers shift from one activity to the next with more or less awareness and explicit demarcation. To treat such con- versational activities as a sequence of discrete units is a convenient oversimplification that is often resorted to (Bokaei et al., 2015; Galley et al., 2003; Passon- neau and Litman, 1997). Systems that provide auto- mated access to spoken language data often rely on segmentation of spoken discourse into sequential units for summarization (Wang and Cardie, 2012; Dielmann and Renals, 2005) or information retrieval (Ward et al., 2015). Research on the organization of spoken dis- course also relies directly or indirectly on identifica- tion of such units to detect agreement among partici- pants (Hillard et al., 2003; Somasundaran et al., 2007; Germesin and Wilson, 2009), multiparty meeting ac- tion items (Purver et al., 2007), decisions (Fern´andez et al., 2008), or answers to questions (Sun and Chai, 2007; Bosma, 2005). To support such research, there is a need for annotation methods to segment conversa- tional interaction into sequential, multi-utterance units. We present and compare two methods to derive such data from crowdsourced annotations. Crowdsourced annotation, where each item is la- beled by a crowd of many independent annotators, is becoming more common in natural language process- ing. Examples include word sense (Bruce and Wiebe, 1999; Snow et al., 2008; Passonneau and Carpenter, 2014), named entities (Finin et al., 2010), and sev- eral other tasks in (Snow et al., 2008), including tex- tual entailment. Three advantages to corpus annotation through application of a probabilistic …

Content truncated to 500 words