Modeling topic dependencies in semantically coherent text spans with copulas

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1767–1776, Osaka, Japan, December 11-17 2016. Modeling topic dependencies in semantically coherent text spans with copulas Georgios Balikas ∗ Université Grenoble-Alpes Computer Science Laboratory Hesam Amoualian Université Grenoble-Alpes Computer Science Laboratory Marianne Clausel Université Grenoble-Alpes Department of Statistics Eric Gaussier Université Grenoble-Alpes Computer Science Laboratory Massih-Reza Amini Université Grenoble-Alpes Computer Science Laboratory Abstract The exchangeability assumption in topic models like Latent Dirichlet Allocation (LDA) often results in inferring inconsistent topics for the words of text spans like noun-phrases, which are usually expected to be topically coherent. We propose copulaLDA, that extends LDA by integrating part of the text structure to the model and relaxes the conditional independence assumption between the word-specific latent topics given the per-document topic distributions. To this end, we assume that the words of text spans like noun-phrases are topically bound and we model this dependence with copulas. We demonstrate empirically the effectiveness of copulaLDA on both intrinsic and extrinsic evaluation tasks on several publicly available corpora. 1 Introduction Probabilistic topic models, such as Latent Dirichlet Allocation (LDA) (Blei et al., 2003), are generative models that describe the content of documents by discovering the latent topics underlying them. A limitation inherent from the bag-of-words representation in such state-of-the-art models concerns the independence assumption: given their topics, words are assumed to occur independently. While this exchangeability assumption greatly impacts the involved computations and, in particular, the calculations of the conditional probabilities, it is rather naive and unrealistic (Heinrich, 2005). As another limitation caused by the exchangeability assumption, the grouping of words in topically coherent spans, that is contiguous text spans like sentences, is lost. On the other hand, text structure generally contains useful information that could be leveraged in inference process. Sentences or phrases, for instance, are by definition text spans complete in themselves that convey a concise statement. To better illustrate how text structure could help in topic identification, consider the example of Figure 1. It illustrates the topics inferred by LDA for the words (excluding stop-words) of a sentence drawn from a Wikipedia page. At the sentence level, one could argue that the sentence is generated by the “Cinema” topic since it discusses a film and its authors. LDA, however, fails and assigns several topics to the words of the sentence. Importantly, several of those topics like “Elections” and “Inventions” are unrelated. In finer text granularity, LDA also fails to assign consistent topics in noun-phrases like “film noir classic” and entities like “Brian Donlevy”. A binding mechanism among the topics of the words of a sentence, or a phrase, could have prevented those limitations and taking simple text structure into account would be beneficial. The film is a remake of the 1947 film noir classic that starred Victor Mature, Brian Donlevy and Richard Widmark. Cinema Science Elections Inventions Figure 1: Applying LDA on Wikipedia documents. Motivated by the previous example, we propose to incorporate text structure in the form of sentence or …

Content truncated to 500 words