Scaling to Large³ Data: An Efficient and Effective Method to Compute Distributional Thesauri

Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 884–890, Seattle, Washington, USA, 18-21 October 2013. c⃝2013 Association for Computational Linguistics Scaling to Large3 Data: An efficient and effective method to compute Distributional Thesauri Martin Riedl and Chris Biemann FG Language Technology Computer Science Department, Technische Universit¨at Darmstadt Hochschulstrasse 10, D-64289 Darmstadt, Germany {riedl,biem}@cs.tu-darmstadt.de Abstract We introduce a new highly scalable approach for computing Distributional Thesauri (DTs). By employing pruning techniques and a dis- tributed framework, we make the computation for very large corpora feasible on comparably small computational resources. We demon- strate this by releasing a DT for the whole vo- cabulary of Google Books syntactic n-grams. Evaluating against lexical resources using two measures, we show that our approach pro- duces higher quality DTs than previous ap- proaches, and is thus preferable in terms of speed and quality for large corpora. 1 Introduction Using larger data to estimate models for machine learning applications as well as for applications of Natural Language Processing (NLP) has repeatedly shown to be advantageous, see e.g. (Banko and Brill, 2001; Brants et al., 2007). In this work, we tackle the influence of corpus size for building a distributional thesaurus (Lin, 1998). Especially, we shed light on the interaction of similarity measures and corpus size, as well as aspects of scalability. We shortly introduce the JoBimText framework for distributional semantics and show its scalability for large corpora. For the computation of the data we follow the MapReduce (Dean and Ghemawat, 2004) paradigm. The computation of similarities between terms becomes challenging on large cor- pora, as both the numbers of terms to be compared and the number of context features increases. This makes standard similarity calculations as proposed in (Lin, 1998; Curran, 2002; Lund and Burgess, 1996; Weeds et al., 2004) computationally infeasi- ble. These approaches first calculate an informa- tion measure between each word and the accord- ing context and then calculate the similarity between all words, based on the information measure for all shared contexts. 2 Related Work A variety of approaches to compute DTs have been proposed to tackle issues regarding size and run- time. The reduction of the feature space seems to be one possibility, but still requires the computation of such reduction cf. (Blei et al., 2003; Golub and Kahan, 1965). Other approaches use randomised in- dexing for storing counts or hashing functions to ap- proximate counts and measures (Gorman and Cur- ran, 2006; Goyal et al., 2010; Sahlgren, 2006). An- other possibility is the usage of distributed process- ing like MapReduce. In (Pantel et al., 2009; Agirre et al., 2009) a DT is computed using MapReduce on 200 quad core nodes (for 5.2 billion sentences) respectively 2000 cores (1.6 Terawords), an amount of hardware only available to commercial search en- gines. Whereas Agirre uses a χ2 test to measure the information between terms and context, Pantel uses the Pointwise Mutual Information (PMI). Then, both approaches use the cosine similarity to calculate the similarity between terms. Furthermore, …

Content truncated to 500 words