SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 722–731 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 722 SP-10K: A Large-scale Evaluation Set for Selectional Preference Acquisition Hongming Zhang, Hantian Ding, and Yangqiu Song Department of CSE, HKUST hzhangal@cse.ust.hk, hdingab@connect.ust.hk, yqsong@cse.ust.hk Abstract Selectional Preference (SP) is a commonly ob- served language phenomenon and proved to be useful in many natural language processing tasks. To provide a better evaluation method for SP models, we introduce SP-10K, a large- scale evaluation set that provides human rat- ings for the plausibility of 10,000 SP pairs over five SP relations, covering 2,500 most fre- quent verbs, nouns, and adjectives in Ameri- can English. Three representative SP acquisi- tion methods based on pseudo-disambiguation are evaluated with SP-10K. To demonstrate the importance of our dataset, we investi- gate the relationship between SP-10K and the commonsense knowledge in ConceptNet5 and show the potential of using SP to represent the commonsense knowledge. We also use the Winograd Schema Challenge to prove that the proposed new SP relations are essential for the hard pronoun coreference resolution problem. 1 Introduction Selectional Preference (SP) is a common phe- nomenon in human language that has been shown to be related to semantics (Wilks, 1975). Here by SP we mean that, given a word and a dependency relation, human beings have preferences for which words are likely to be connected. For instance, when seeing the verb ‘sing’, it is highly plausi- ble that its object is ‘a song’, and when seeing the noun ‘air’, it is highly plausible that its modifier is ‘fresh’. SP has been shown to be useful over a variety of tasks including sense disambiguation (Resnik, 1997), semantic role classification (Zapirain et al., 2013), coreference clustering (Hobbs, 1978; Inoue et al., 2016; Heinzerling et al., 2017), and ma- chine translation (Tang et al., 2016). Given the importance of SP, the automatic acquisition of SP has become a well-known research subject in the SP Evaluation Set #R #W #P (McRae et al., 1998) 2 641 821 (Keller and Lapata, 2003) 3 571 540 (Pad´o et al., 2006) 3 180 207 SP-10K 5 2.5K 10K Table 1: Statistics of Human-labeled SP Evaluation Sets. #R, #W, and #P indicate the number of SP re- lation types, words, and pairs, respectively. NLP community. However, current SP acquisition models are limited based on existing evaluation methods. We discuss two broadly used evalua- tion methods, human-labeled evaluation sets and the pseudo-disambiguation task. First, the most straightforward way to evalu- ate SP models is by asking human annotators. McRae et al. (1998), Keller and Lapata (2003), and Pad´o et al. (2006) proposed human-labeled SP evaluation sets containing hundreds of SP pairs (numbers are shown in Table 1). However, these datasets are too small to cover the diversity of the SP task adequately. Moreover, they only consid- ered one-hop relations, such as ‘verb-object’ and ‘modifier-noun’ pairs. Aside from these relations, we believe that higher-order dependency relations may also …

Content truncated to 500 words