Comparing Approaches for Automatic Question Identification

Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (*SEM 2017), pages 110–114, Vancouver, Canada, August 3-4, 2017. c⃝2017 Association for Computational Linguistics Comparing Approaches for Automatic Question Identification Angel Samsuddin Maredia, Kara Schechtman, Sarah Ita Levitan, Julia Hirschberg Department of Computer Science, Columbia University, USA asm2221@columbia.edu, kws2121@columbia.edu, sarahita@cs.columbia.edu, julia@cs.columbia.edu Abstract Collecting spontaneous speech corpora that are open-ended, yet topically con- strained, is increasingly popular for re- search in spoken dialogue systems and speaker state, inter alia. Typically, these corpora are labeled by human annota- tors, either in the lab or through crowd- sourcing; however, this is cumbersome and time-consuming for large corpora. We present four different approaches to auto- matically tagging a corpus when general topics of the conversations are known. We develop these approaches on the Columbia X-Cultural Deception corpus and find ac- curacy that significantly exceeds the base- line. Finally, we conduct a cross-corpus evaluation by testing the best performing approach on the Columbia/SRI/Colorado corpus. 1 Introduction Corpora of spontaneous speech are often col- lected through interviews or by otherwise provid- ing subjects with question prompts. Such cor- pora are semi-structured; they are constrained by the prompts used, but the elicited speech is open- ended in vocabulary and structure. It is often de- sirable to segment these corpora into their under- lying topics based on the questions asked. This is typically done manually by annotators in the lab or via crowd-sourcing. However, such annotation is impractical and time-consuming for large corpora. In this paper we describe a set of experi- ments aimed at automatically tagging a large cor- pus with topic labels. We tag the Columbia X- Cultural Deception (CXD) corpus, a large-scale (120-hour) corpus of deceptive and non-deceptive dialogues collected using a semi-structured inter- view paradigm. Participants took turns interview- ing each other using a fixed set of biographical in- terview questions1, but the questions were asked in individual variants, in any order, and interview- ers often asked follow-up questions. For example, the question, ”Are your parents divorced?” could be produced as ”Are your mom and dad still to- gether?” These questions are semantically simi- lar, but differ lexically, presenting the challenge of topically tagging a corpus based on semantic similarity. The question, ”Have you ever broken a bone?” could be followed by another, ”How did you break your bone?” This illustrates the chal- lenge of distinguishing between phrases that are lexically similar, but differ semantically. These two examples highlight problems faced when try- ing to automatically annotate a corpus for re- sponses to a given set of questions. With such a large corpus, it is not practical to manually annotate topic boundaries. So, to com- pare question responses from multiple subjects, we identify conversational turns in the corpus that correspond to the original interview questions. We compare four approaches to question identifica- tion: (1) a baseline approach that identifies ques- tions using strict string matches, (2) the ROUGE metric which is based on n-gram comparisons, (3) cosine similarity between word embedding repre- sentations …

Content truncated to 500 words