Quality Assessment of Large Scale Knowledge Resources

Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP 2006), pages 534–541, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Quality Assessment of Large Scale Knowledge Resources Montse Cuadros IXA NLP Group EHU/UPV Donostia, Basque Country mcuadros001@ikasle.ehu.es German Rigau IXA NLP Group EHU/UPV Donostia, Basque Country german.rigau@ehu.es Abstract This paper presents an empirical eval- uation of the quality of publicly avail- able large-scale knowledge resources. The study includes a wide range of manu- ally and automatically derived large-scale knowledge resources. In order to establish a fair and neutral comparison, the qual- ity of each knowledge resource is indi- rectly evaluated using the same method on a Word Sense Disambiguation task. The evaluation framework selected has been the Senseval-3 English Lexical Sample Task. The study empirically demonstrates that automatically acquired knowledge re- sources surpass both in terms of preci- sion and recall the knowledge resources derived manually, and that the combina- tion of the knowledge contained in these resources is very close to the most frequent sense classifier. As far as we know, this is the first time that such a quality assessment has been performed showing a clear pic- ture of the current state-of-the-art of pub- licly available wide coverage semantic re- sources. 1 Introduction Using large-scale semantic knowledge bases, such as WordNet (Fellbaum, 1998), has become a usual, often necessary, practice for most current Natural Language Processing systems. Even now, building large and rich enough knowledge bases for broad–coverage semantic processing takes a great deal of expensive manual effort involving large research groups during long periods of de- velopment. This fact has severely hampered the state-of-the-art of current Natural Language Pro- cessing (NLP) applications. For example, dozens of person-years have been invested in the develop- ment of wordnets for various languages (Vossen, 1998), but the data in these resources seems not to be rich enough to support advanced concept-based NLP applications directly. It seems that applica- tions will not scale up to working in open domains without more detailed and rich general-purpose (and also domain-specific) linguistic knowledge built by automatic means. For instance, in more than eight years of man- ual construction (from version 1.5 to 2.0), Word- Net passed from 103,445 semantic relations to 204,074 semantic relations1. That is, around twelve thousand semantic relations per year. How- ever, during the last years the research commu- nity has devised a large set of innovative processes and tools for large-scale automatic acquisition of lexical knowledge from structured or unstructured corpora. Among others we can mention eX- tended WordNet (Mihalcea and Moldovan, 2001), large collections of semantic preferences acquired from SemCor (Agirre and Martinez, 2001; Agirre and Martinez, 2002) or acquired from British Na- tional Corpus (BNC) (McCarthy, 2001), large- scale Topic Signatures for each synset acquired from the web (Agirre and de la Calle, 2004) or acquired from the BNC (Cuadros et al., 2005). Obviously, all these semantic resources have been acquired using a very different set of meth- ods, tools and corpora, resulting on a …

Content truncated to 500 words