Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 1739–1746 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 1739 Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collections Yi-An Lai†1, Xuan Zhu†1, Yi Zhang1, Mona Diab∗2 1Amazon AWS AI 2The George Washington University {yianl, zhuxuan, yizhngn}@amazon.com, mtdiab@gwu.edu Abstract Summarizing data samples by quantitative measures has a long history, with descriptive statistics being a case in point. However, as natural language processing methods flourish, there are still insufficient characteristic metrics to describe a collection of texts in terms of the words, sentences, or paragraphs they comprise. In this work, we propose metrics of diversity, density, and homogeneity that quantitatively measure the dispersion, sparsity, and uniformity of a text collection. We conduct a series of simulations to verify that each metric holds desired properties and resonates with human intuitions. Experiments on real-world datasets demonstrate that the proposed characteristic metrics are highly correlated with text classification performance of a renowned model, BERT, which could inspire future applications. Keywords: diversity, density, homogeneity, characteristics, text classification, quantitative measures 1. Introduction Characteristic metrics are a set of unsupervised measures that quantitatively describe or summarize the properties of a data collection. These metrics generally do not use ground- truth labels and only measure the intrinsic characteristics of data. The most prominent example is descriptive statis- tics that summarizes a data collection by a group of un- supervised measures such as mean or median for central tendency, variance or minimum-maximum for dispersion, skewness for symmetry, and kurtosis for heavy-tailed anal- ysis. In recent years, text classification, a category of Natural Language Processing (NLP) tasks, has drawn much at- tention (Zhang et al., 2015; Joulin et al., 2016; Howard and Ruder, 2018) for its wide-ranging real-world applica- tions such as fake news detection (Shu et al., 2017), doc- ument classification (Yang et al., 2016), and spoken lan- guage understanding (SLU) (Gupta et al., 2019a; Gupta et al., 2019b; Zhang et al., 2018), a core task of conversational assistants like Amazon Alexa or Google Assistant. However, there are still insufficient characteristic metrics to describe a collection of texts. Unlike numeric or categorical data, simple descriptive statistics alone such as word counts and vocabulary size are difficult to capture the syntactic and semantic properties of a text collection. In this work, we propose a set of characteristic metrics: di- versity, density, and homogeneity to quantitatively summa- rize a collection of texts where the unit of texts could be a phrase, sentence, or paragraph. A text collection is first mapped into a high-dimensional embedding space. Our characteristic metrics are then computed to measure the dis- persion, sparsity, and uniformity of the distribution. Based on the choice of embedding methods, these characteristic metrics can help understand the properties of a text col- lection from different linguistic perspectives, for example, †These authors contributed equally. ∗Work done as a Principal Scientist at AWS AI. lexical diversity, syntactic variation, and semantic homo- geneity. Our proposed diversity, …

Content truncated to 500 words