Probabilistic Domain Modelling With Contextualized Distributional Semantic Vectors

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 392–401, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Probabilistic Domain Modelling With Contextualized Distributional Semantic Vectors Jackie Chi Kit Cheung University of Toronto 10 King’s College Rd., Room 3302 Toronto, ON, Canada M5S 3G4 jcheung@cs.toronto.edu Gerald Penn University of Toronto 10 King’s College Rd., Room 3302 Toronto, ON, Canada M5S 3G4 gpenn@cs.toronto.edu Abstract Generative probabilistic models have been used for content modelling and template induction, and are typically trained on small corpora in the target domain. In contrast, vector space models of distribu- tional semantics are trained on large cor- pora, but are typically applied to domain- general lexical disambiguation tasks. We introduce Distributional Semantic Hidden Markov Models, a novel variant of a hid- den Markov model that integrates these two approaches by incorporating contex- tualized distributional semantic vectors into a generative model as observed emis- sions. Experiments in slot induction show that our approach yields improvements in learning coherent entity clusters in a do- main. In a subsequent extrinsic evalua- tion, we show that these improvements are also reflected in multi-document summa- rization. 1 Introduction Detailed domain knowledge is crucial to many NLP tasks, either as an input for language un- derstanding, or as the goal itself, to acquire such knowledge. For example, in information extrac- tion, a list of slots in the target domain is given to the system, and in natural language generation, content models are trained to learn the content structure of texts in the target domain for infor- mation structuring and automatic summarization. Generative probabilistic models have been one popular approach to content modelling. An impor- tant advantage of this approach is that the structure of the model can be adapted to fit the assumptions about the structure of the domain and the nature of the end task. As this field has progressed, the formal structures that are assumed to represent a domain have increased in complexity and become more hierarchical. Earlier work assumes a flat set of topics (Barzilay and Lee, 2004), which are ex- pressed as states of a latent random variable in the model. Later work organizes topics into a hierar- chy from general to specific (Haghighi and Van- derwende, 2009; Celikyilmaz and Hakkani-Tur, 2010). Recently, Cheung et al. (2013) formalized a domain as a set of frames consisting of proto- typical sequences of events, slots, and slot fillers or entities, inspired by classical AI work such as Schank and Abelson’s (1977) scripts. We adopt much of this terminology in this work. For exam- ple, in the CRIMINAL INVESTIGATIONS domain, there may be events such as a murder, an investi- gation of the crime, an arrest, and a trial. These would be indicated by event heads such as kill, ar- rest, charge, plead. Relevant slots would include VICTIM, SUSPECT, AUTHORITIES, PLEA, etc. One problem faced by this line of work is that, by their nature, these models are typically trained on a small corpus from the …

Content truncated to 500 words