Hidden Markov tree models for semantic class induction

Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pages 94–103, Sofia, Bulgaria, August 8-9 2013. c⃝2013 Association for Computational Linguistics Hidden Markov tree models for semantic class induction ´Edouard Grave Inria - Sierra Project-Team ´Ecole Normale Sup´erieure Paris, France Edouard.Grave @inria.fr Guillaume Obozinski Universit´e Paris-Est, LIGM ´Ecole des Ponts - ParisTech Marne-la-Vall´ee, France Guillaume.Obozinski @imagine.enpc.fr Francis Bach Inria - Sierra Project-Team ´Ecole Normale Sup´erieure Paris, France Francis.Bach @ens.fr Abstract In this paper, we propose a new method for semantic class induction. First, we in- troduce a generative model of sentences, based on dependency trees and which takes into account homonymy. Our model can thus be seen as a generalization of Brown clustering. Second, we describe an efficient algorithm to perform inference and learning in this model. Third, we apply our proposed method on two large datasets (108 tokens, 105 words types), and demonstrate that classes induced by our algorithm improve performance over Brown clustering on the task of semi- supervised supersense tagging and named entity recognition. 1 Introduction Most competitive learning methods for compu- tational linguistics are supervised, and thus re- quire labeled examples, which are expensive to obtain. Moreover, those techniques suffer from data scarcity: many words only appear a small number of time, or even not at all, in the training data. It thus helps a lot to first learn word clus- ters on a large amount of unlabeled data, which are cheap to obtain, and then to use this clusters as features for the supervised task. This scheme has proven to be effective for various tasks such as named entity recognition (Freitag, 2004; Miller et al., 2004; Liang, 2005; Faruqui et al., 2010), syntactic chunking (Turian et al., 2010) or syntac- tic dependency parsing (Koo et al., 2008; Haffari et al., 2011; Tratz and Hovy, 2011). It was also successfully applied for transfer learning of multi- lingual structure by T¨ackstr¨om et al. (2012). The most commonly used clustering method for semi-supervised learning is the one proposed by Brown et al. (1992), and known as Brown clus- tering. While still being one of the most efficient word representation method (Turian et al., 2010), Brown clustering has two limitations we want to address in this work. First, since it is a hard clus- tering method, homonymy is ignored. Second, it does not take into account syntactic relations be- tween words, which seems crucial to induce se- mantic classes. Our goal is thus to propose a method for semantic class induction which takes into account both syntax and homonymy, and then to study their effects on semantic class learning. In this paper, we start by introducing a new un- supervised method for semantic classes induction. This is achieved by defining a generative model of sentences with latent variables, which aims at capturing semantic roles of words. We require our method to be scalable, in order to learn models on large datasets containing tens of millions of sen- tences. More precisely, we make the following contributions: • We introduce …

Content truncated to 500 words