Inherited Feature-based Similarity Measure Based on Large Semantic Hierarchy and Large Text Corpus

Inherited Feature-based Similarity Measure Based on Large Semantic Hierarchy and Large Text Corpus Hideki Hirakawa Toshiba I{.&D Center 1 Komukai Toshiba-cho, Saiwai-ku, Kawasaki 210, JAPAN hirakawa¢ee:l., rdc. toshiba, co. j p Zhonghui Xu, Kenneth Haase MIT Media Laboratory 20 Ames Street Cambridge, MA 02139 USA {xu, haase} @media. mit. edu Abstract We describe a similarity calculation model called IFSM (Inherited Feature Similarity Measure) between objects (words/concepts) based on their com- mon and distinctive features. We pro- pose an implementation method for ob- taining features based on abstracted triples extracted fi'om a large text eorpus utilizing taxonomical knowledge. This model represents an integration of tradi- tional methods, i.e,. relation b~used sin> itarity measure and distribution based similarity measure. An experiment, us- ing our new concept abstraction method which we <'all the fiat probability group- ing method, over 80,000 surface triples, shows that the abstraction level of 3000 is a good basis for feature description. '1 Introduction Determination of semantic similarity between words is an important component of linguis- tic tasks ranging from text retrieval and filter- ing, word sense disambiguation or text match- ing. In the past five years, this work has evolved in conjunction with the availability of powerful computers and large linguistic resources such as WordNet (Miller,90), the EDR concept dictionary (EDR,93), and large text corpora. Similarity methods can be broadly divided into "relation based" methods which use relations in an ontology to determine similarity and "distribu- tion based" methods which use statistical analysis as the basis of similarity judgements. This article describes a new method of similarity nmtehing, in- herited feature based similarity matching (IFSM) which integrates these two approaches. Relation based methods include both depth based and path based measures of similarity. The Most Specific Common Abstraction (MSCA) method compares two concepts based on the tax- onomic depth of their common parent; for exam- ple, "dolphin" and "human" are more similar than "oak" and "human" because the common concept "mammal" is deeper in the taxonomy than "living thing". Path-length similarity methods are based on counting the links between nodes in a semantic network. (Rada,89) is a widely adopted approach to such matching and (Sussna,93) combines it with WordNet to do semantic disambiguation. The chief problems with relation-b~sed similar- ity methods lie in their sensitivity to artifacts in the coding of the ontology, l;or instance, MSCA algorithms are sensitive to the relative deplh and detail of different parts of the concept taxon- omy. If one conceptual domain (say plants) is sketchily represented while another conceptual do- main (say,, animals) is richly represented, similar- ity comparisons within the two domains will be in- commensurable. A similar problem plagues path- length based algorithms, causing nodes in richly structured parts of the ontology to be consistently judged less similm" to one another than nodes in shallower or hess complete parts of the ontology. Distribution-based methods are based on the idea that the similarity of words can be derived frorn the similarity of the contexts in which they occur. These methods difl'er …

Content truncated to 500 words