Diverging Divergences: Examining Variants of Jensen Shannon Divergence for Corpus Comparison Tasks

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6740–6744 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6740 Diverging Divergences: Examining Variants of Jensen Shannon Divergence for Corpus Comparison Tasks Jinghui Lu, Maeve Henchion, Brian Mac Namee Insight Centre for Data Analytics, Teagasc Agriculture and Food Development Authority Dublin Ireland, Dublin Ireland Jinghui.Lu@ucdconnect.ie, Maeve.henchion@teagasc.ie, Brian.MacNamee@ucd.ie Abstract Jensen-Shannon divergence (JSD) is a distribution similarity measurement widely used in natural language processing. In corpus comparison tasks, where keywords are extracted to reveal the divergence between different corpora (for example, social media posts from proponents of different views on a political issue), two variants of JSD have emerged in the literature. One of these uses a weighting based on the relative sizes of the corpora being compared. In this paper we argue that this weighting is unnecessary and, in fact, can lead to misleading results. We recommend that this weighted version is not used. We base this recommendation on an analysis of the JSD variants and experiments showing how they impact corpus comparison results as the relative sizes of the corpora being compared change. Keywords: Corpus Comparison, Jensen-Shannon Divergence 1. Introduction Corpus comparison approaches are widely used in the dig- ital humanities to identify the differences between corpora through identification of divergent keywords. A symmetric divergence metric, Jensen-Shannon divergence (JSD) (Lin, 1991), is one of the most frequently applied techniques used to identify divergent keywords in this type of task. For example, JSD has been used to compare social media posts from different social groups (Gallagher et al., 2016; Lu et al., 2017; Mangold, 2016) or articles from pairs of differ- ent years (Pechenick, 2015; Koplenig, 2015; Gerlach et al., 2016; Woodward III, 2016). When this approach to corpus comparison is used, diver- gence scores for each word in the two corpora being com- pared are calculated by JSD and the k words with the high- est divergence scores are usually shown to a user, along with the corpus within which they are most prevalent. Visu- alisations such as those shown in Figure 1 are often used for this. In each back-to-back bar chart, we list the top 30 most divergent terms found within two corpora. Higher ranks in- dicate more distinguishing words, and the directions of the bars indicate the corpus within which a word is more preva- lent. For example, in Figure 1(c) which shows the results of a corpus comparison between articles from two different sections of the New York Times, a bar to the left indicates a term is more common in the education section while a bar to the right suggests that a term is more common in the sports section. Two versions of JSD for performing corpus comparison have emerged in the literature (Pechenick et al., 2015a; Gal- lagher et al., 2016). In this paper we argue, however, that these two variants give dramatically different, and in some cases misleading results, and so more care should be …

Content truncated to 500 words