Evaluating Roget‘s Thesauri

Proceedings of ACL-08: HLT, pages 416–424, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Evaluating Roget’s Thesauri Alistair Kennedy School of Information Technology and Engineering University of Ottawa Ottawa, Ontario, Canada akennedy@site.uottawa.ca Stan Szpakowicz School of Information Technology and Engineering University of Ottawa Ottawa, Ontario, Canada and Institute of Computer Science Polish Academy of Sciences Warsaw, Poland szpak@site.uottawa.ca Abstract Roget’s Thesaurus has gone through many re- visions since it was first published 150 years ago. But how do these revisions affect Ro- get’s usefulness for NLP? We examine the differences in content between the 1911 and 1987 versions of Roget’s, and we test both ver- sions with each other and WordNet on prob- lems such as synonym identification and word relatedness. We also present a novel method for measuring sentence relatedness that can be implemented in either version of Roget’s or in WordNet. Although the 1987 version of the Thesaurus is better, we show that the 1911 ver- sion performs surprisingly well and that often the differences between the versions of Ro- get’s and WordNet are not statistically signif- icant. We hope that this work will encourage others to use the 1911 Roget’s Thesaurus in NLP tasks. 1 Introduction Roget’s Thesaurus, first introduced over 150 years ago, has gone through many revisions to reach its current state. We compare two versions, the 1987 and 1911 editions of the Thesaurus with each other and with WordNet 3.0. Roget’s Thesaurus has a unique structure, quite different from WordNet, of which the NLP community has yet to take full ad- vantage. In this paper we demonstrate that although the 1911 version of the Thesaurus is very old, it can give results comparable to systems that use WordNet or newer versions of Roget’s Thesaurus. The main motivation for working with the 1911 Thesaurus instead of newer versions is that it is in the public domain, along with related NLP-oriented software packages. For applications that call for an NLP-friendly thesaurus, WordNet has become the de-facto standard. Although WordNet is a fine re- sources, we believe that ignoring other thesauri is a serious oversight. We show on three applications how useful the 1911 Thesaurus is. We ran the well- established tasks of determining semantic related- ness of pairs of terms and identifying synonyms (Jar- masz and Szpakowicz, 2004). We also proposed a new method of representing the meaning of sen- tences or other short texts using either WordNet or Roget’s Thesaurus, and tested it on the data set pro- vided by Li et al. (2006). We hope that this work will encourage others to use Roget’s Thesaurus in their own NLP tasks. Previous research on the 1987 version of Roget’s Thesaurus includes work of Jarmasz and Szpakow- icz (2004). They propose a method of determin- ing semantic relatedness between pairs of terms. Terms that appear closer together in the Thesaurus get higher weights than those farther apart. The experiments aimed at identifying synonyms using a modified version of the proposed semantic sim- ilarity function. …

Content truncated to 500 words