Do we Need Linguistics When We Have Statistics? A Comparative Analysis of the Contributions of Linguistic Cues to a Statistical Word Grouping System

DO WE NEED LINGUISTICS WHEN WE HAVE STATISTICS? A COMPARATIVE ANALYSIS OF THE CONTRIBUTIONS OF LINGUISTIC CUES TO A STATISTICAL WORD GROUPING SYSTEM Vasileios Hatzivassiloglou Deparlrnent of Computer Science 450 Computer Science Building Columbia University New York, N.Y. 10027 vh@cs.columbia.edu ABSTRACT We present a comparative analysis of the performance of a statistics-based system for the formation of semantic groups of adjectives when various sources of linguistic knowledge are introduced. We identify four different types of slufllow linguistic knowledge that are applicable to this system, and we quantify the performance gained by incorporating each such knowledge module, We perform experiments for dif- ferent corpus sizes and different inputs (sets of adjectives to group), collect clam on the usaful.ness of each linguistic module, assess the statistical significance of the results, and compare the contributions of the linguistic knowledge sources against each other. We also assess the overall effect linguistic knowledge has in our system. Our results show that linguistic knowledge causes a significant increase in the performance of the system. We conclude by discussing how these positive restdts can be generalized to other problems in statistical NLP. 1. INTRODUCTION The idea of integrating statistical and knowledge-based approaches for natural language problems, has been. recently. . gainin$ ground, in. the computational lingmsucs commumty, as it is ex- pected that a combined approach will offer sig- nificantly better performance over either methodology alone. This paper supplements this intuitive belief with actual evaluatzon data, ob- tained when several linguistics-based modules were integrated in a statistical system. We used a system we previously developed for the separation of adjectives into semantic groups [Hatzivassiloglou and McKeown, 1993] as the basis for our comparative analysis. We identified several different types of shallow linguistic knowledge that can be efficiently introduced into our system. We evaluated the system with and 43 without each such feature, obtaining an estimate of each feature's positive or negative contribution to the overall performance. By matching cases where all system parameters are the same except for one teature, we assess the statistical significance of the differences found. Also, a statistical model of the system's performance in terms of the active fea- tures for each run offers a view of the contribu- tions of features from a different angle, contrasting the significance of linguistic features (or other modeled system parameters) against each other. Our analysis of the experimental results showed that many forms of li%.uistic knowledge have a significant positive conmbution to the per- formance of the system. We attribute to the com- bined effect of the linguistic knowledge modules the ability of our system to perform fine-tuned classification of adjectives into semantic classes. Other statistical systems that address word clas- sification probleans do not emphasize the use of linguistic knowledge and do not deal with a specific word class[Brown et al., 1992], or do not exploit as much linguistic knowledge as we do [Pereira et al., 1993]. As a result, a coarser clas- sification is usually produced. In contrast, by limiting the system's input to …

Content truncated to 500 words