Novel Association Measures Using Web Search with Double Checking

Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the ACL, pages 1009–1016, Sydney, July 2006. c⃝2006 Association for Computational Linguistics Novel Association Measures Using Web Search with Double Checking Hsin-Hsi Chen Ming-Shun Lin Yu-Chuan Wei Department of Computer Science and Information Engineering National Taiwan University Taipei, Taiwan hhchen@csie.ntu.edu.tw;{mslin,ycwei}@nlg.csie.ntu.edu.tw Abstract A web search with double checking model is proposed to explore the web as a live corpus. Five association measures including variants of Dice, Overlap Ratio, Jaccard, and Cosine, as well as Co- Occurrence Double Check (CODC), are presented. In the experiments on Ruben- stein-Goodenough’s benchmark data set, the CODC measure achieves correlation coefficient 0.8492, which competes with the performance (0.8914) of the model using WordNet. The experiments on link detection of named entities using the strategies of direct association, associa- tion matrix and scalar association matrix verify that the double-check frequencies are reliable. Further study on named en- tity clustering shows that the five meas- ures are quite useful. In particular, CODC measure is very stable on word- word and name-name experiments. The application of CODC measure to expand community chains for personal name dis- ambiguation achieves 9.65% and 14.22% increase compared to the system without community expansion. All the experi- ments illustrate that the novel model of web search with double checking is fea- sible for mining associations from the web. 1 Introduction In statistical natural language processing, re- sources used to compute the statistics are indis- pensable. Different kinds of corpora have made available and many language models have been experimented. One major issue behind the cor- pus-based approaches is: if corpora adopted can reflect the up-to-date usage. As we know, lan- guages are live. New terms and phrases are used in daily life. How to capture the new usages is an important research topic. The Web is a heterogeneous document collec- tion. Huge-scale and dynamic nature are charac- teristics of the Web. Regarding the Web as a live corpus becomes an active research topic re- cently. How to utilize the huge volume of web data to measure association of information is an important issue. Resnik and Smith (2003) em- ploy the Web as parallel corpora to provide bi- lingual sentences for translation models. Keller and Lapata (2003) show that bigram statistics for English language is correlated between corpus and web counts. Besides, how to get the word counts and the word association counts from the web pages without scanning over the whole col- lections is indispensable. Directly managing the web pages is not an easy task when the Web grows very fast. Search engine provides some way to return useful information. Page counts for a query de- note how many web pages containing a specific word or a word pair roughly. Page count is dif- ferent from word frequency, which denotes how many occurrences a word appear. Lin and Chen (2004) explore the use of the page counts pro- vided by different search engines to compute the statistics for Chinese segmentation. …

Content truncated to 500 words