Evaluating the Inferential Utility of Lexical-Semantic Resources

Proceedings of the 12th Conference of the European Chapter of the ACL, pages 558–566, Athens, Greece, 30 March – 3 April 2009. c⃝2009 Association for Computational Linguistics Evaluating the Inferential Utility of Lexical-Semantic Resources Shachar Mirkin, Ido Dagan, Eyal Shnarch Computer Science Department, Bar-Ilan University Ramat-Gan 52900, Israel {mirkins,dagan,shey}@cs.biu.ac.il Abstract Lexical-semantic resources are used ex- tensively for applied semantic inference, yet a clear quantitative picture of their current utility and limitations is largely missing. We propose system- and application-independent evaluation and analysis methodologies for resources’ per- formance, and systematically apply them to seven prominent resources. Our find- ings identify the currently limited recall of available resources, and indicate the po- tential to improve performance by exam- ining non-standard relation types and by distilling the output of distributional meth- ods. Further, our results stress the need to include auxiliary information regarding the lexical and logical contexts in which a lexical inference is valid, as well as its prior validity likelihood. 1 Introduction Lexical information plays a major role in seman- tic inference, as the meaning of one term is of- ten inferred form another. Lexical-semantic re- sources, which provide the needed knowledge for lexical inference, are commonly utilized by ap- plied inference systems (Giampiccolo et al., 2007) and applications such as Information Retrieval and Question Answering (Shah and Croft, 2004; Pasca and Harabagiu, 2001). Beyond WordNet (Fell- baum, 1998), a wide range of resources has been developed and utilized, including extensions to WordNet (Moldovan and Rus, 2001; Snow et al., 2006) and resources based on automatic distri- butional similarity methods (Lin, 1998; Pantel and Lin, 2002). Recently, Wikipedia is emerg- ing as a source for extracting semantic relation- ships (Suchanek et al., 2007; Kazama and Tori- sawa, 2007). As of today, only a partial comparative picture is available regarding the actual utility and limi- tations of available resources for lexical-semantic inference. Works that do provide quantitative information regarding resources utility have fo- cused on few particular resources (Kouylekov and Magnini, 2006; Roth and Sammons, 2007) and evaluated their impact on a specific system. Most often, works which utilized lexical resources do not provide information about their isolated con- tribution; rather, they only report overall per- formance for systems in which lexical resources serve as components. Our paper provides a step towards clarify- ing this picture. We propose a system- and application-independent evaluation methodology that isolates resources’ performance, and sys- tematically apply it to seven prominent lexical- semantic resources. The evaluation and analysis methodology is specified within the Textual En- tailment framework, which has become popular in recent years for modeling practical semantic infer- ence in a generic manner (Dagan and Glickman, 2004). To that end, we assume certain definitions that extend the textual entailment paradigm to the lexical level. The findings of our work provide useful insights and suggested directions for two research com- munities: developers of applied inference systems and researchers addressing lexical acquisition and resource construction. Beyond the quantitative mapping of resources’ performance, our analysis points at issues concerning …

Content truncated to 500 words