Measuring Correlation Between Linguist’s Judgments and Latent Dirichlet Allocation Topics

Measuring Correlation Between Linguists’ Judgments and Latent Dirichlet Allocation Topics Ari Chanen and Jon Patrick School of Information Technologies University of Sydney Sydney, Australia, 2007 {ari,jonpat}@it.usyd.edu.au Abstract Data that has been annotated by linguists is often considered a gold standard on many tasks in the NLP field. However, linguists are expensive so researchers seek automatic techniques that correlate well with human performance. Linguists working on the ScamSeek project were given the task of deciding how many and which document classes existed in this previously unseen cor- pus. This paper investigates whether the document classes identified by the linguists correlate significantly with Latent Dirichlet Allocation (LDA) topics induced from that corpus. Monte-Carlo simulation is used to measure the statistical significance of the correlation between LDA models and the linguists’ characterisations. In experiments, more than 90% of the linguists’ classes met the level required to declare the correlation between linguistic insights and LDA models is significant. These results help verify the usefulness of the LDA model in NLP and are a first step in showing that the LDA model can replace the efforts of linguists in certain tasks like subdividing a corpus into classes. 1 Introduction Since linguists are expensive to employ, there is a preference in most NLP projects to use automatic processes especially where it can be shown that the automatic process approaches the performance of the linguists. Several linguists were used on the ScamSeek project (Patrick, 2006). ScamSeek was created for the Australian Securities and Investments Commission (ASIC) government agency to identify financial scam websites based on the linguistic prop- erties of the webpage content. A major task they performed by the project linguists was to partition the corpus into classes. Besides defining the classes in terms of the documents assigned to them, the lin- guists also identified phrases they believed were in- dicative of each class. The LDA corpus model (Blei, 2004) can automat- ically generate a likely set of corpus topics and sub- divide the corpus words among those topics. We will show that there are similarities between the task the LDA performs and the tasks the ScamSeek lin- guists performed. This paper attempts to determine to what degree LDA topics correlate with the judg- ments of linguists in partitioning a corpus into doc- ument classes. Formally, we set a null hypothesis, H0, to claim that the relationship between the linguists’ docu- ment classes and LDA topics is random. The al- ternative hypothesis, Ha, claims those document classes and the topics have a significant amount of correspondence or correlation between them. In or- der to measure how significant the correlation is, principled methods of measuring the statistical sig- nificance of the correlation must be found. If the p- value for the correlation between a document class and the best correlating topic for that class is less than α = 0.05, then H0 will be rejected in favor of Ha. The determination of the p-values are discussed in the Methods section. Proceedings of the Australasian Language Technology Workshop …

Content truncated to 500 words