Efficient induction of probabilistic word classes with LDA

Proceedings of the 5th International Joint Conference on Natural Language Processing, pages 363–372, Chiang Mai, Thailand, November 8 – 13, 2011. c⃝2011 AFNLP Efficient induction of probabilistic word classes with LDA Grzegorz Chrupała Spoken Language Systems Saarland University gchrupala@lsv.uni-saarland.de Abstract Word classes automatically induced from distributional evidence have proved use- ful many NLP tasks including Named En- tity Recognition, parsing and sentence re- trieval. The Brown hard clustering algo- rithm is commonly used in this scenario. Here we propose to use Latent Dirichlet Allocation in order to induce soft, prob- abilistic word classes. We compare our approach against Brown in terms of ef- ficiency. We also compare the useful- ness of the induced Brown and LDA word classes for the semi-supervised learning of three NLP tasks: fine-grained Named En- tity Recognition, Morphological Analysis and semantic Relation Classification. We show that using LDA for word class in- duction scales better with the number of classes than the Brown algorithm and the resulting classes outperform Brown on the three tasks. 1 Introduction Word classes automatically induced from distri- butional evidence have proved useful in a vari- ety of tasks, including Named Entity Recogni- tion (Miller et al. 2004, Ratinov and Roth 2009, Chrupała and Klakow 2010, Turian et al. 2010), parsing (Koo et al. 2008, Suzuki et al. 2009, Candito and Crabb´e 2009) and sentence retrieval (Momtazi and Klakow 2009). Brown et al. (1992) introduced an algorithm which assigns word types to disjoint clusters and it remains a common choice when a simple way to automatically obtain word classes is needed. We present a word class induction method using La- tent Dirichlet Allocation (Blei et al. 2003) which has attractive properties compared to Brown: • It induces a soft, probabilistic clustering on both word types and context features. • It runs in time linear in the number of classes. The model maps straightforwardly to the standard document topic model, and thus has the advan- tage of many existing high quality implementa- tions. We evaluate the model’s usefulness on fine- grained Named Entity Recognition (NER), Mor- phological Analysis (MA) and semantic Relation Classification (RC) and show that • while the word classes obtained perform bet- ter than Brown classes, • they can be induced in a fraction of the time necessary to run the equivalent Brown model. 2 Inducing word representations There is a variety of approaches to inducing word representations from distributional information. In this section we briefly review the research most relevant to our proposed approach. Hard classes Brown et al. (1992) introduced an early model which induces a mapping from word types to classes. It is an agglomerative clustering algorithm which starts with K classes for the K most frequent word types and then proceeds by al- ternately adding the next most frequent word to the class set and merging the two classes which result in the least decrease in the mutual informa- tion between class bigrams. The result is a class hierarchy with word types at the leaves. The …

Content truncated to 500 words