A Mixed-Method Design Approach for Empirically Based Selection of Unbiased Data Annotators

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 186–192 August 1–6, 2021. ©2021 Association for Computational Linguistics 186 More than just Frequency? Demasking Unsupervised Hypernymy Prediction Methods Thomas Bott, Dominik Schlechtweg and Sabine Schulte im Walde Institute for Natural Language Processing, University of Stuttgart, Germany {bottts,schlecdk,schulte}@ims.uni-stuttgart.de Abstract This paper presents a comparison of unsuper- vised methods of hypernymy prediction (i.e., to predict which word in a pair of words such as fish–cod is the hypernym and which the hy- ponym). Most importantly, we demonstrate across datasets for English and for German that the predictions of three methods (Weeds- Prec, invCL, SLQS Row) strongly overlap and are highly correlated with frequency-based predictions. In contrast, the second-order method SLQS shows an overall lower accu- racy but makes correct predictions where the others go wrong. Our study once more con- firms the general need to check the frequency bias of a computational method in order to identify frequency-(un)related effects. 1 Introduction Hypernymy represents a major paradigmatic se- mantic relation between two concepts, a hypernym (superordinate) and a hyponym (subordinate), as in tree–oak and fish–cod, where the hyponym implies the hypernym, but not vice versa. From a cognitive perspective hypernymy is central to the organisa- tion of the mental lexicon (Deese, 1965; Miller and Fellbaum, 1991; Murphy, 2003), next to further semantic relations such as synonymy, antonymy, etc. From a computational perspective hypernymy is central to solving a number of Natural Language Processing (NLP) tasks such as taxonomy creation (Hearst, 1998; Cimiano et al., 2004; Snow et al., 2006; Navigli and Ponzetto, 2012), textual entail- ment (Dagan et al., 2006; Clark et al., 2007) and text generation (Biran and McKeown, 2013). Accordingly, the field has witnessed active re- search on two subtasks involved in computational models of hypernymy (see Shwartz et al. (2017) for an extensive overview): hypernymy detection (i.e., distinguishing hypernymy from other semantic rela- tions) and hypernymy prediction (i.e., determining which word in a pair of words is the hypernym and which is the hyponym). The target subtask of the current study is hypernymy prediction: we perform a comparative analysis of a class of ap- proaches commonly refered to as unsupervised hy- pernymy methods (Weeds et al., 2004; Kotlerman et al., 2010; Clarke, 2012; Lenci and Benotto, 2012; Santus et al., 2014). These methods all rely on the distributional hypothesis (Harris, 1954; Firth, 1957) that words which are similar in meaning also occur in similar linguistic distributions. In this vein, they exploit asymmetries in distributional vector space representations, in order to contrast hypernym and hyponym vectors. While these unsupervised hypernymy prediction methods have been explored and compared exten- sively on a number of benchmark datasets (Shwartz et al., 2017), this study takes a novel perspective and performs a detailed analysis of whether and where the methods make similar or different deci- sions. Our prediction experiments on simplex and complex nouns in English and German WordNets and evaluation benchmarks show that most of the methods we investigate overlap in their specific …

Content truncated to 500 words