Undersampling Improves Hypernymy Prototypicality Learning

Undersampling Improves Hypernymy Prototypicality Learning Koki Washio, Tsuneaki Kato Department of Language and Information Sciences The University of Tokyo 3-8-1, Komaba, Meguroku, Tokyo 153-8902 Japan {kokiwashio@g.ecc, kato@boz.c}.u-tokyo.ac.jp Abstract This paper focuses on supervised hypernymy detection using distributional representations for unknown word pairs. Levy et al. (2015) demonstrated that supervised hypernymy detection suffers from overfitting hypernyms in training data. We show that the problem of overfitting on this task is caused by a characteristic of datasets, which stems from the inherent structure of the language resources used, hierarchical thesauri. The simple data preprocessing method proposed in this paper alleviates this problem. To be more precise, we demonstrate through experiments that the problem that hypernymy classifiers overfit hypernyms in training data comes from a skewed word frequency distribution brought by the quasi-tree structure of a thesaurus, which is a major resource of lexical semantic relation data, and propose a simple undersampling method based on word frequencies that can effectively alleviate overfitting and improve distributional prototypicality learning for unknown word pairs. Keywords: lexical semantic relations, hypernymy detection, taxonomy induction 1. Introduction Detecting hypernymy relations between unknown words contributes to Taxonomy Induction, which induces a tax- onomy in a new domain (Panchenko et al., 2016). Supervised distributional hypernymy detection represents each word pair (x, y) as combined distributional represen- tations, and trains a classifier that discriminates based on whether the word pair has a relation. Frequently used meth- ods for combining word representations include vector con- catenation ⃗x ⊕ ⃗y (CONCAT) and difference ⃗y − ⃗x (DIFF) (Baroni et al., 2012; Fu et al., 2014; Roller et al., 2014; Weeds et al., 2014; Vylomova et al., 2016). The supervised methods have been reported to be better than the distribu- tional unsupervised measures (Roller et al., 2014; Weeds et al., 2014). However, Levy et al. (2015) demonstrated that supervised classifiers have some problems. Two major problems are as follows: Problem 1 Classifiers do not learn relations in word pairs but only learn distributional prototypicality at best. Problem 2 Classifiers overfit hypernyms, especially those in training data (lexical memorization). Distributional prototypicality, if learned correctly, is still useful. Shwartz et al. (2016) integrated this information into their neural path-based model, which captures rela- tions between two words, and improved the performance significantly. Roller and Erk (2016) demonstrated that the prototypicality learned by CONCAT captures Hearst pat- terns such as ”fruits such as apples.” Their method using the CONCAT model as a feature detector has high general- ization performance. Thus, resolving Problem 2 is expected to improve the performance of the previous models. In this paper, we investigate why classifiers overfit hyper- nyms in training data. We analyze this problem from the point of view of the skewed distribution of frequencies of hypernyms in training data, which stems from the inher- ent structure of language resources such as hierarchical the- sauri. We show that an imbalance of word frequencies ad- versely affects classifiers, and we verify our analysis by ex- periments. Moreover, we show that a simple undersampling …

Content truncated to 500 words