Nine Features in a Random Forest to Learn Taxonomical Semantic Relations

Nine Features in a Random Forest to Learn Taxonomical Semantic Relations Enrico Santus*, Alessandro Lenci§, Tin-Shing Chiu*, Qin Lu*, Chu-Ren Huang* * The Hong Kong Polytechnic University, Hong Kong esantus@gmail.com, cstschiu@comp.polyu.edu.hk, {qin.lu, churen.huang}@polyu.edu.hk § University of Pisa, Italy alessandro.lenci@unipi.it Abstract ROOT9 is a supervised system for the classification of hypernyms, co-hyponyms and random words that is derived from the already introduced ROOT13 (Santus et al., 2016). It relies on a Random Forest algorithm and nine unsupervised corpus-based features. We evaluate it with a 10-fold cross validation on 9,600 pairs, equally distributed among the three classes and involving several Parts-Of-Speech (i.e. adjectives, nouns and verbs). When all the classes are present, ROOT9 achieves an F1 score of 90.7%, against a baseline of 57.2% (vector cosine). When the classification is binary, ROOT9 achieves the following results against the baseline: hypernyms-co-hyponyms 95.7% vs. 69.8%, hypernyms-random 91.8% vs. 64.1% and co-hyponyms-random 97.8% vs. 79.4%. In order to compare the performance with the state-of-the-art, we have also evaluated ROOT9 in subsets of the Weeds et al. (2014) datasets, proving that it is in fact competitive. Finally, we investigated whether the system learns the semantic relation or it simply learns the prototypical hypernyms, as claimed by Levy et al. (2015). The second possibility seems to be the most likely, even though ROOT9 can be trained on negative examples (i.e., switched hypernyms) to drastically reduce this bias. Keywords: Vector Space Models, VSMs, Distributional Semantic Models, DSMs, Semantic Relations, Taxonomy, Hypernymy, Entailment, Hyponymy, Co-Hyponymy 1. Introduction Distinguishing hypernyms from co-hyponyms and, in turn, discriminating them from semantically unrelated words (henceforth randoms) is a fundamental task in Natural Language Processing (NLP). Hypernymy in fact represents a key organization principle of semantic memory (Murphy, 2002), the backbone of taxonomies and ontologies, and one of the crucial semantic relations supporting lexical entailment (Geffet and Dagan, 2005). Co-hyponymy (or coordination) is instead the relation held by words sharing a close hypernym, which are therefore attributionally similar (Weeds et al., 2014). The ability of discriminating hypernymy, co-hyponymy and random words has potentially infinite applications, including Automatic Thesauri Creation, Paraphrasing, Textual Entailment, Sentiment Analysis and so on (Weeds et al., 2014; Tungthamthiti et al. 2015). For this reason, in the last decades, numerous methods, datasets and shared tasks have been proposed to improve computers’ ability in such discrimination, generally achieving promising results (Santus et al., 2016b; Roller et al., 2014, Weeds et al., 2014; Santus et al. 2014a; Rimmel, 2014; Lenci and Benotto, 2012; Kotlerman et al., 2010; Geffet and Dagan, 2005; Weeds and Weir, 2003). Both supervised and unsupervised approaches have been investigated. The former have been shown to outperform the latter in Weeds et al. (2014), even though Levy et al. (2015) have claimed that these methods may learn whether a term y is a prototypical hypernym, regardless of its actual relation with a term x. In this paper we further investigate and revise ROOT13 (Santus et al., 2016b), a supervised method based on a Random Forest algorithm and thirteen corpus-based features. …

Content truncated to 500 words