A Large-Scale Corpus for Conversation Disentanglement

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3846–3856 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3846 A Large-Scale Corpus for Conversation Disentanglement Jonathan K. Kummerfeld1∗ Sai R. Gouravajhala1 Joseph J. Peper1 Vignesh Athreya1 Chulaka Gunasekara2 Jatin Ganhotra2 Siva Sankalp Patel2 Lazaros Polymenakos2 Walter S. Lasecki1 Computer Science & Engineering1 T.J. Watson Research Center2 University of Michigan IBM Research AI Abstract Disentangling conversations mixed together in a single stream of messages is a difficult task, made harder by the lack of large manually an- notated datasets. We created a new dataset of 77,563 messages manually annotated with reply-structure graphs that both disentangle conversations and define internal conversation structure. Our dataset is 16 times larger than all previously released datasets combined, the first to include adjudication of annotation dis- agreements, and the first to include context. We use our data to re-examine prior work, in particular, finding that 80% of conversations in a widely used dialogue corpus are either miss- ing messages or contain extra messages. Our manually-annotated data presents an opportu- nity to develop robust data-driven methods for conversation disentanglement, which will help advance dialogue research. 1 Introduction When a group of people communicate in a com- mon channel there are often multiple conversa- tions occurring concurrently. Often there is no ex- plicit structure identifying conversations or their structure, such as in Internet Relay Chat (IRC), Google Hangout, and comment sections on web- sites. Even when structure is provided it often has limited depth, such as threads in Slack, which provide one layer of branching. In all of these cases, conversations are entangled: all messages appear together, with no indication of separate conversations. Automatic disentanglement could be used to provide more interpretable results when searching over chat logs, and to help users under- stand what is happening when they join a chan- nel. Over a decade of research has considered conversation disentanglement (Shen et al., 2006), but using datasets that are either small (2,500 mes- sages, Elsner and Charniak, 2008) or not released (Adams and Martell, 2008). ∗ jkummerf@umich.edu We introduce a conversation disentanglement dataset of 77,563 messages of IRC manually anno- tated with reply-to relations between messages.1 Our data is sampled from a technical support chan- nel at 173 points in time between 2004 and 2018, providing a diverse set of speakers and topics, while remaining in a single domain. Our data is the first to include context, which differentiates messages that start a conversation from messages that are responding to an earlier point in time. We are also the first to adjudicate disagreements in disentanglement annotations, producing higher quality development and test sets. We also devel- oped a simple model that is more effective than prior work, and showed that having diverse data makes it perform better and more consistently. We also analyze prior disentanglement work. In particular, a recent approach from Lowe et al. (2015, 2017). By applying disentanglement to an enormous log of IRC messages, …

Content truncated to 500 words