Automatic Thesaurus Construction for Modern Hebrew

Automatic Thesaurus Construction for Modern Hebrew °Chaya Liebeskind, *Ido Dagan, *Jonathan Schler °Computer Science Department, Jerusalem College of Technology, Lev Academic Center, Israel *Computer Science Department, Bar-Ilan University, Israel liebchaya@gmail.com, dagan@cs.biu.ac.il, schler@gmail.com Abstract Automatic thesaurus construction for Modern Hebrew is a complicated task, due to its high degree of inflectional ambiguity. Linguistics tools, including morphological analyzers, part-of-speech taggers and parsers often have limited in performance on Morphologically Rich Languages (MRLs) such as Hebrew. In this paper, we adopted a schematic methodology for generating a co- occurrence based thesaurus in a MRL and extended the methodology to create distributional similarity thesaurus. We explored three alternative levels of morphological term representations, surface form, lemma, and multiple lemmas, all complemented by the clustering of morphological variants. First, we evaluated both the co-occurrence based method and the distributional similarity method using Hebrew WordNet as our gold standard. However, due to Hebrew WordNet's low coverage, we completed our analysis with a manual evaluation. The results showed that for Modern Hebrew corpus-based thesaurus construction, the most directly applied statistical collection, using linguistics tools at the lemma level, is not optimal. Keywords: Hebrew, Morphologically Rich Language, thesaurus, co-occurrence, distributional similarity 1. Introduction and Background A thesaurus is a lexical resource which groups words together by semantic similarity. The notion of semantic similarity includes semantic relations, such as synonyms (car-automobile), hyperonyms (vehicle-car), hyponyms (car-vehicle), meronyms (wheel-car), and antonyms (acceleration-deceleration). Generally, two statistical approaches for corpus-based thesaurus construction were explored: a first-order, co- occurrence-based approach which assumes that words that frequently occur together are topically related (Schütze and Pedersen 1997) and a second-order, distributional similarity approach (Hindle 1990; Lin 1998; Gasperin et al. 2001; Weeds and Weir 2003; Kotlerman et al. 2010) which suggests that words occurring within similar contexts are semantically similar (Harris 1968). The aim of this research was to construct a thesaurus for Modern Hebrew. Modern Hebrew lacks repositories of machine-readable knowledge (i.e, lexical resources) fundamental to many Natural Language Processing (NLP) tasks, such as machine translation (Irvine and Callison- Burch 2013), question answering (Yang et al. 2015), and word sense disambiguation (Chen et al. 2014). Therefore, in the last decade, a few semantic resources for modern Hebrew have been constructed. Examples for such resources are Hebrew WordNet1 (Ordan et al. 2007), which groups words into sets of synonyms, and Hebrew FrameNet (Hayoun and Elhadad 2016), which defines formal structures for semantic frames, and various relationships between and within them. Furthermore, Hebrew is a Morphologically Rich Language (MRL), a language in which significant information about syntactic units and relations is expressed at word-level. Due to the rich and challenging morphology of MRLs, tools, such as part-of-speech taggers, and parsers, often perform poorly. In MRLs, commonly used statistical extraction at the lemma level, using a morphological analyzer and tagger (Lindén and Piitulainen 2004; Peirsman et al. 2008; Rapp 2008), might not be the optimal choice. 1 http://cl.haifa.ac.il/projects/mwn/index.shtml Therefore, Liebeskind et al. (2012) suggested a methodology for generating a co-occurrence based thesaurus in MRL. They explored three options …

Content truncated to 500 words