SemEval-2007 Task 16: Evaluation of Wide Coverage Knowledge Resources

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 81–86, Prague, June 2007. c⃝2007 Association for Computational Linguistics SemEval-2007 Task 16: Evaluation of Wide Coverage Knowledge Resources Montse Cuadros TALP Research Center Universitat Polit´ecnica de Catalunya Barcelona, Spain cuadros@lsi.upc.edu German Rigau IXA NLP Group Euskal Herriko Unibersitatea Donostia, Spain german.rigau@ehu.es Abstract This task tries to establish the relative qual- ity of available semantic resources (derived by manual or automatic means). The qual- ity of each large-scale knowledge resource is indirectly evaluated on a Word Sense Dis- ambiguation task. In particular, we use Senseval-3 and SemEval-2007 English Lex- ical Sample tasks as evaluation bechmarks to evaluate the relative quality of each re- source. Furthermore, trying to be as neu- tral as possible with respect the knowledge bases studied, we apply systematically the same disambiguation method to all the re- sources. A completely different behaviour is observed on both lexical data sets (Senseval- 3 and SemEval-2007). 1 Introduction Using large-scale knowledge bases, such as Word- Net (Fellbaum, 1998), has become a usual, often necessary, practice for most current Natural Lan- guage Processing (NLP) systems. Even now, build- ing large and rich enough knowledge bases for broad–coverage semantic processing takes a great deal of expensive manual effort involving large re- search groups during long periods of development. In fact, dozens of person-years have been invested in the development of wordnets for various languages (Vossen, 1998). For example, in more than ten years of manual construction (from version 1.5 to 2.1), WordNet passed from 103,445 semantic relations to 245,509 semantic relations1. That is, around one thousand new relations per month. But this data does not seems to be rich enough to support ad- vanced concept-based NLP applications directly. It seems that applications will not scale up to work- ing in open domains without more detailed and rich general-purpose (and also domain-specific) seman- tic knowledge built by automatic means. Fortunately, during the last years, the research community has devised a large set of innovative methods and tools for large-scale automatic acqui- sition of lexical knowledge from structured and un- structured corpora. Among others we can men- tion eXtended WordNet (Mihalcea and Moldovan, 2001), large collections of semantic preferences ac- quired from SemCor (Agirre and Martinez, 2001; Agirre and Martinez, 2002) or acquired from British National Corpus (BNC) (McCarthy, 2001), large- scale Topic Signatures for each synset acquired from the web (Agirre and de la Calle, 2004) or acquired from the BNC (Cuadros et al., 2005). Obviously, these semantic resources have been acquired using a very different set of methods, tools and corpora, re- sulting on a different set of new semantic relations between synsets (or between synsets and words). Many international research groups are working on knowledge-based WSD using a wide range of ap- proaches (Mihalcea, 2006). However, less attention has been devoted on analysing the quality of each semantic resource. In fact, each resource presents different volume and accuracy figures (Cuadros et al., 2006). In this paper, we evaluate those resources on …

Content truncated to 500 words