Disentangling Chat with Local Coherence Models

Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, pages 1179–1189, Portland, Oregon, June 19-24, 2011. c⃝2011 Association for Computational Linguistics Disentangling Chat with Local Coherence Models Micha Elsner School of Informatics University of Edinburgh melsner0@gmail.com Eugene Charniak Department of Computer Science Brown University, Providence, RI 02912 ec@cs.brown.edu Abstract We evaluate several popular models of local discourse coherence for domain and task gen- erality by applying them to chat disentangle- ment. Using experiments on synthetic multi- party conversations, we show that most mod- els transfer well from text to dialogue. Co- herence models improve results overall when good parses and topic models are available, and on a constrained task for real chat data. 1 Introduction One property of a well-written document is coher- ence, the way each sentence �ts into its context– sen- tences should be interpretable in light of what has come before, and in turn make it possible to inter- pret what comes after. Models of coherence have primarily been used for text-based generation tasks: ordering units of text for multidocument summariza- tion or inserting new text into an existing article. In general, the corpora used consist of informative writing, and the tasks used for evaluation consider different ways of reordering the same set of textual units. But the theoretical concept of coherence goes beyond both this domain and this task setting– and so should coherence models. This paper evaluates a variety of local coher- ence models on the task of chat disentanglement or “threading”: separating a transcript of a multiparty interaction into independent conversations1. Such simultaneous conversations occur in internet chat 1A public implementation is available via https:// bitbucket.org/melsner/browncoherence. rooms, and on shared voice channels such as push- to-talk radio. In these situations, a single, correctly disentangled, conversational thread will be coherent, since the speakers involved understand the normal rules of discourse, but the transcript as a whole will not be. Thus, a good model of coherence should be able to disentangle sentences as well as order them. There are several differences between disentan- glement and the newswire sentence-ordering tasks typically used to evaluate coherence models. Inter- net chat comes from a different domain, one where topics vary widely and no reliable syntactic annota- tions are available. The disentanglement task mea- sures different capabilities of a model, since it com- pares documents that are not permuted versions of one another. Finally, full disentanglement requires a large-scale search, which is computationally dif- �cult. We move toward disentanglement in stages, carrying out a series of experiments to measure the contribution of each of these factors. As an intermediary between newswire and inter- net chat, we adopt the SWITCHBOARD (SWBD) cor- pus. SWBD contains recorded telephone conversa- tions with known topics and hand-annotated parse trees; this allows us to control for the performance of our parser and other informational resources. To compare the two algorithmic settings, we use SWBD for ordering experiments, and also arti�cially entan- gle pairs of telephone dialogues to create synthetic transcripts which we …

Content truncated to 500 words