Evaluating a Topic Modelling Approach to Measuring Corpus Similarity

Evaluating a Topic Modelling Approach to Measuring Corpus Similarity Richard Fothergill,♦ Paul Cook♣ and Timothy Baldwin♦ ♦ Department of Computing and Information Systems The University of Melbourne Victoria 3010, Australia ♣ Faculty of Computer Science University of New Brunswick Fredericton, NB E3B 5A3, Canada rjfo@unimelb.edu.au, paul.cook@unb.ca, tb@ldwin.net Abstract Web corpora are often constructed automatically, and their contents are therefore often not well understood. One technique for assessing the composition of such a web corpus is to empirically measure its similarity to a reference corpus whose composition is known. In this paper we evaluate a number of measures of corpus similarity, including a method based on topic modelling which has not been previously evaluated for this task. To evaluate these methods we use known-similarity corpora that have been previously used for this purpose, as well as a number of newly-constructed known-similarity corpora targeting differences in genre, topic, time, and region. Our findings indicate that, overall, the topic modelling approach did not improve on a chi-square method that had previously been found to work well for measuring corpus similarity. Keywords: Analyzing corpora, corpus similarity, topic modelling 1. Introduction In constructing traditional corpora, such as the British Na- tional Corpus (Burnard, 2000, BNC), documents are cho- sen based on particular selection criteria such as domain, genre, and time period. The composition of such a cor- pus in terms of these factors is therefore understood. On the other hand, many web corpora are constructed automat- ically on the basis of web crawls (Ferraresi et al., 2008), or the results of search engine queries (Baroni and Bernar- dini, 2004). We therefore don’t have the same understand- ing of the composition of the corpora that are built, and the size of such corpora generally precludes manual analysis of their composition. This provided the underlying motivation for this paper: can we develop automatic analytic methods to help gain a better understanding of the composition of automatically-constructed corpora, such as web corpora? One way to analyze a corpus is to measure the extent to which it is similar to other corpora, in particular reference corpora which we do know the composition of. Unfortu- nately, other than Kilgarriff (2001), there has been very lit- tle work to-date on this topic. Kilgarriff (2001) analyzed a number of empirical methods for measuring corpus simi- larity and found a method based on the chi-square statistic (“χ2”) to perform best. In this paper we consider an al- ternative approach to measuring corpus similarity based on topic modelling (Blei et al., 2003). Kilgarriff (2001) cautioned that corpus similarity is com- plex, noting that two corpora can be similar in some ways and different in others, and that a single measure of cor- pus similarity is therefore limited. However, because of the lack of prior work in this area, Kilgarriff (2001) argued a measure of corpus similarity to be a useful starting point. In this paper we further explore the types of difference be- tween corpora that a variety of corpus similarity measures are able …

Content truncated to 500 words