Identifying Comparable Corpora Using LDA

2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 558–562, Montr´eal, Canada, June 3-8, 2012. c⃝2012 Association for Computational Linguistics Identifying Comparable Corpora Using LDA Judita Preiss j.preiss@sheffield.ac.uk Department of Computer Science, University of Sheffield, Regent Court 211 Portobello, Sheffield, S1 4DP, United Kingdom Abstract Parallel corpora have applications in many ar- eas of Natural Language Processing, but are very expensive to produce. Much information can be gained from comparable texts, and we present an algorithm which, given any bod- ies of text in multiple languages, uses ex- isting named entity recognition software and topic detection algorithm to generate pairs of comparable texts without requiring a paral- lel corpus training phase. We evaluate the system’s performance firstly on data from the online newspaper domain, and secondly on Wikipedia cross-language links. 1 Introduction Manual alignment or creation of parallel corpora is exceedingly expensive, requiring highly skilled an- notators or professional translators. Methods exist for aligning parallel corpora, and extracted parallel segments can be used to, for example, augment ma- chine translation phrase tables, but the amount of genuinely parallel data is limited. However, paral- lel segments can also be extracted from comparable corpora (a comparable corpus is one which contains similar texts in more than one language). Compara- ble documents, if produced with a confidence value, could also be used to prioritize translation (manual or automatic) when one is searching for further in- formation (which may only be available in a foreign language) to augment information given in an arti- cle in the source language. We present a technique to automatically detect comparable corpora in exist- ing data, and we demonstrate the applicability of our method to any genre by evaluating on crawled online newspaper text, as well as Wikipedia articles. Clearly, texts need to contain some of the same data in order to be comparable (Harris, 1954), and we assume: • To be similar, texts need to share some named entities, e.g., T´oth et al., (2008). • Comparable texts need to be on the same topic. Construction of multilingual topic models usu- ally requires either parallel data or some number of aligned documents across multiple languages. Zhao and Xing (2007) create bilingual topic models from (at least 25%) of parallel data. Mimno et al., (2009) start from tuples of equivalent documents to build models, and then the same distribution over topics holds in both source and target languages. While Zhao and Xing (2007) used their topic models for word alignment from comparable cor- pora (combined with underlying parallel data), mul- tilingual topic models are usually applied to data to automatically detect word translations based on par- allel data, e.g., Vuli´c et al., (2011) exploit a shared language independent topic distribution to measure the similarity between topics pertaining to words. The novelty of our work is the transformation of a source language topic model rather than the creation of a language independent model from parallel data. Transforming the source language model to the …

Content truncated to 500 words