The importance of high-quality input for WSD: an application-oriented comparison of part-of-speech taggers

The Importance of High Quality Input for WSD: An Application-Oriented Comparison of Part-of-Speech Taggers Tanja Gaustad Humanities Computing University of Groningen P.O. Box 716 9700 AS Groningen, The Netherlands Tel +31-(0)50-363 59 77 Fax +31-(0)50-363 68 55 T.Gaustad@let.rug.nl Abstract In this paper, we present an application- oriented evaluation of three Part-of- Speech (PoS) taggers in a word sense disambiguation (WSD) system. Follow- ing the intuition that high quality input is likely to influence the final results of a complex system, we test whether the more accurate taggers also produce better re- sults when integrated into the WSD sys- tem. For this purpose, a stand-alone eval- uation of the PoS taggers is used to assess which tagger is the most accurate. The re- sults of the WSD task, computed on the training section of the Dutch Senseval-2 data, including the PoS information from all three taggers show that the most ac- curate PoS tags do indeed lead to the best results, thereby verifying our hypoth- esis. A surprising result, however, is the fact that the performance of the complex WSD system with the different PoS tags included does not necessarily reflect the stand-alone accuracy of the PoS taggers. 1 Introduction Certain NLP tools are typically used as a sub- component or a pre-processor in a more complex system, rather than as a complete application in their own right. A typical example of such tools are Part- of-Speech (PoS) taggers. What is usually not taken into account is the fact that the quality (in terms of accuracy) of each subpart of a complex system is likely to influence the final results considerably. Lately, standardized evaluation of NLP resources has gained more importance in the field of Compu- tational Linguistics (e.g. CLEF workshops in infor- mation retrieval, Parseval, Senseval), but a tendency towards more application-oriented evaluation is only beginning. In this paper, we will proceed to an application- oriented comparison of three PoS taggers in a word sense disambiguation (WSD) system. We will eval- uate to what extent differences in stand-alone PoS accuracy influence the results obtained in the com- plex WSD system using the acquired PoS informa- tion. Since the Dutch data we use is not only am- biguous with regard to meaning but also with regard to PoS, accurate PoS information is very important to achieve high disambiguation accuracy. The paper is structured as follows: We will start with a detailed description and comparison of the three PoS taggers including a stand-alone evalua- tion in order to compare their performance indepen- dently of the application to the WSD task. Then fol- lows a description of the WSD system in which (the output of) the different PoS taggers will be incorpo- rated and tested. This includes a presentation of the machine learning algorithm employed for classifica- tion (maximum entropy) and its application to WSD, as well as a note on the data and the settings used for the reported experiments. Next, the application- dependent results of the three PoS taggers will be …

Content truncated to 500 words