Statistical Thesaurus Construction for a Morphologically Rich Language

First Joint Conference on Lexical and Computational Semantics (*SEM), pages 59–64, Montr´eal, Canada, June 7-8, 2012. c⃝2012 Association for Computational Linguistics Statistical Thesaurus Construction for a Morphologically Rich Language Chaya Liebeskind, Ido Dagan and Jonathan Schler Computer Science Department Bar-Ilan University Ramat-Gan, Israel liebchaya@gmail.com, dagan@cs.biu.ac.il, schler@gmail.com Abstract Corpus-based thesaurus construction for Mor- phologically Rich Languages (MRL) is a com- plex task, due to the morphological variability of MRL. In this paper we explore alternative term representations, complemented by cluster- ing of morphological variants. We introduce a generic algorithmic scheme for thesaurus con- struction in MRL, and demonstrate the empiri- cal benefit of our methodology for a Hebrew thesaurus. 1 Introduction Corpus-based thesaurus construction has been an active research area (Grefenstette, 1994; Curran and Moens, 2002; Kilgarriff, 2003; Rychly and Kilgarriff, 2007). Typically, two statistical ap- proaches for identifying semantic relationships between words were investigated: first-order, co- occurrence-based methods which assume that words that occur frequently together are topically related (Schutze and Pederson, 1997) and second- order, distributional similarity methods (Hindle, 1990; Lin, 1998; Gasperin et al, 2001; Weeds and Weir, 2003; Kotlerman et al., 2010), which suggest that words occurring within similar contexts are semantically similar (Harris, 1968). While most prior work focused on English, we are interested in applying these methods to MRL. Such languages, Hebrew in our case, are character- ized by highly productive morphology which may produce as many as thousands of word forms for a given root form. Thesauri usually provide related terms for each entry term (denoted target term). Since both target and related terms correspond to word lemmas, sta- tistics collection from the corpus would be most directly applied at the lemma level as well, using a morphological analyzer and tagger (Linden and Piitulainen, 2004; Peirsman et al., 2008; Rapp, 2009). However, due to the rich and challenging morphology of MRL, such tools often have limited performance. In our research, the accuracy of a state-of-the-art modern Hebrew tagger on a cross genre corpus was only about 60%. Considering such limited performance of mor- phological processing, we propose a schematic methodology for generating a co-occurrence based thesaurus in MRL. In particular, we propose and investigate three options for term representation, namely surface form, lemma and multiple lemmas, supplemented with clustering of term variants. While the default lemma representation is depend- ent on tagger performance, the two other represen- tations avoid choosing the right lemma for each word occurrence. Instead, the multiple-lemma rep- resentation assumes that the right analysis will ac- cumulate enough statistical prominence throughout the corpus, while the surface representation solves morphological disambiguation "in retrospect", by clustering term variants at the end of the extraction process. As the methodology provides a generic scheme for exploring the alternative representation levels, each corpus and language-specific tool set might yield a different optimal configuration. 2 Methodology Thesauri usually contain thousands of entries, termed here target terms. Each entry holds a list of related terms, covering various semantic relations. In this paper we assume that the list of …

Content truncated to 500 words