Lessons from the Bible on Modern Topics: Low-Resource Multilingual Topic Model Evaluation

Proceedings of NAACL-HLT 2018, pages 1090–1100 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Lessons from the Bible on Modern Topics: Low-Resource Multilingual Topic Model Evaluation Shudong Hao Computer Science University of Colorado Boulder, CO shudong@colorado.edu Jordan Boyd-Graber Computer Science, iSchool, LSC, and UMIACS, University of Maryland College Park, MD jbg@umiacs.umd.edu Michael J. Paul Information Science University of Colorado Boulder, CO mpaul@colorado.edu Abstract Multilingual topic models enable document analysis across languages through coherent multilingual summaries of the data. However, there is no standard and effective metric to evaluate the quality of multilingual topics. We introduce a new intrinsic evaluation of multi- lingual topic models that correlates well with human judgments of multilingual topic coher- ence as well as performance in downstream ap- plications. Importantly, we also study evalua- tion for low-resource languages. Because stan- dard metrics fail to accurately measure topic quality when robust external resources are un- available, we propose an adaptation model that improves the accuracy and reliability of these metrics in low-resource settings. 1 Introduction Topic models provide a high-level view of the main themes of a document collection (Boyd-Graber et al., 2017). Document collections, however, are often not in a single language, driving the develop- ment of multilingual topic models. These models discover topics that are consistent across languages, providing useful tools for multilingual text analy- sis (Vuli´c et al., 2015), such as detecting cultural differences (Gutiérrez et al., 2016) and bilingual dictionary extraction (Liu et al., 2015). Monolingual topic models can be evaluated through likelihood (Wallach et al., 2009b) or co- herence (Newman et al., 2010), but topic model evaluation is not well understood in multilingual settings. Our contributions are two-fold. We in- troduce an improved intrinsic evaluation metric for multilingual topic models, called Crosslingual Normalized Pointwise Mutual Information (CNPMI, Section 2). We explore the behaviors of CNPMI at both the model and topic levels with six language pairs and varying model specifications. This metric correlates well with human judgments and crosslin- gual classification results (Sections 5 and 6). We also focus on evaluation in low-resource lan- guages, which lack large parallel corpora, dictio- naries, and other tools that are often used in learn- ing and evaluating topic models. To adapt CNPMI to these settings, we create a coherence estimator (Section 3) that extrapolates statistics derived from antiquated, specialized texts like the Bible: often the only resource available for many languages. 2 Evaluating Multilingual Coherence A multilingual topic contains one topic for each language. For a multilingual topic to be meaning- ful to humans (Figure 1), the meanings should be consistent across the languages, in addition to co- herent within each language (i.e., all words in a topic are related). This section describes our approach to evaluating the quality of multilingual topics. After defining the multilingual topic model, we describe topic model evaluation extending standard monolingual approaches to multilingual settings. 2.1 Multilingual Topic Modeling Probabilistic topic models associate each document in a corpus with a distribution over latent topics, while …

Content truncated to 500 words