Mixed Membership Markov Models for Unsupervised Conversation Modeling

Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 94–104, Jeju Island, Korea, 12–14 July 2012. c⃝2012 Association for Computational Linguistics Mixed Membership Markov Models for Unsupervised Conversation Modeling Michael J. Paul Center for Language and Speech Processing Johns Hopkins University Baltimore, MD 21218, USA mpaul@cs.jhu.edu Abstract Recent work has explored the use of hidden Markov models for unsupervised discourse and conversation modeling, where each seg- ment or block of text such as a message in a conversation is associated with a hidden state in a sequence. We extend this approach to al- low each block of text to be a mixture of mul- tiple classes. Under our model, the probability of a class in a text block is a log-linear func- tion of the classes in the previous block. We show that this model performs well at predic- tive tasks on two conversation data sets, im- proving thread reconstruction accuracy by up to 15 percentage points over a standard HMM. Additionally, we show quantitatively that the induced word clusters correspond to speech acts more closely than baseline models. 1 Introduction The proliferation of social media in recent years has lead to an increased use of informal Web data in the language processing community. With this ris- ing interest in social domains, it is natural to con- sider models which explicitly incorporate the con- versational patterns of social text. Compared to the naive approach of treating conversations as flat doc- uments, models which include conversation struc- ture have been shown to improve tasks such as fo- rum search (Elsas and Carbonell, 2009; Seo et al., 2009), question answering and expert finding (Xu et al., 2008; Wang et al., 2011a), and interpersonal re- lationship identification (Diehl et al., 2007). While conversational features may be important, Web-derived corpora are not always annotated with this information, and the nature of conversations on the Web can vary wildly across domains and venues. Addressing these concerns, there has been recent work with unsupervised models of Web conversa- tions based on hidden Markov models (Ritter et al., 2010), where each state corresponds to a conversa- tional class or “act.” Unlike more traditional uses of HMMs in which a single token is emitted per time step, HMM emissions in conversations correspond to entire blocks of text, such that an entire message is generated at each step. Because each time step is associated with a block of variables, we refer to this type of HMM as a block HMM (Fig. 1a). While block HMMs offer a concise model of inter-message structure, they have the limitation that each text block (message) belongs to exactly one class. Many modern generative models of text, in contrast, allow documents to contain many latent classes. For example, topic models such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003) as- sume each document has its own distribution over multiple classes (often called “topics”). For many predictive tasks, topic models outperform single- class generative …

Content truncated to 500 words