New Experiments in Distributional Representations of Synonymy

Proceedings of the 9th Conference on Computational Natural Language Learning (CoNLL), pages 25–32, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics New Experiments in Distributional Representations of Synonymy Dayne Freitag, Matthias Blume, John Byrnes, Edmond Chow, Sadik Kapadia, Richard Rohwer, Zhiqiang Wang HNC Software, LLC 3661 Valley Centre Drive San Diego, CA 92130, USA � DayneFreitag,MatthiasBlume,JohnByrnes,EdChow, SadikKapadia,RichardRohwer,ZhiqiangWang � @fairisaac.com Abstract Recent work on the problem of detect- ing synonymy through corpus analysis has used the Test of English as a Foreign Lan- guage (TOEFL) as a benchmark. How- ever, this test involves as few as 80 ques- tions, prompting questions regarding the statistical significance of reported results. We overcome this limitation by generating a TOEFL-like test using WordNet, con- taining thousands of questions and com- posed only of words occurring with suf- ficient corpus frequency to support sound distributional comparisons. Experiments with this test lead us to a similarity mea- sure which significantly outperforms the best proposed to date. Analysis suggests that a strength of this measure is its rela- tive robustness against polysemy. 1 Introduction Many text applications are predicated on the idea that shallow lexical semantics can be acquired through corpus analysis. Harris articulated the ex- pectation that words with similar meanings would be used in similar contexts (Harris, 1968), and recent empirical work involving large corpora has borne this out. In particular, by associating each word with a distribution over the words observed in its context, we can distinguish synonyms from non-synonyms with fair reliability. This capability may be ex- ploited to generate corpus-based thesauri automat- ically (Lin, 1998), or used in any other application of text that might benefit from a measure of lexi- cal semantic similarity. And synonymy is a logical first step in a broader research program that seeks to account for natural language semantics through dis- tributional means. Previous research into corpus-analytic approaches to synonymy has used the Test of English as a For- eign Language (TOEFL). The TOEFL consists of 300 multiple-choice question, each question involv- ing five words: the problem or target word and four response words, one of which is a synonym of the target. The objective is to identify the synonym (call this the answer word, and call the other response words decoys). In the context of research into lexi- cal semantics, we seek a distance function which as reliably as possible orders the answer word in front of the decoys. Landauer and Dumais first proposed the TOEFL as a test of lexical semantic similarity and reported a score of 64.4% on an 80-question version of the TOEFL, a score nearly identical to the average score of human test takers (Landauer and Dumais, 1997). Subsequently, Sahlgren reported a score of 72.0% on the same test using “random indexing” and a dif- ferent training corpus (Sahlgren, 2001). By analyz- ing a much larger corpus, Ehlert was able to score 82% on a 300-question version of the TOEFL, using a simple distribution over contextual words (Ehlert, 2003). While success on …

Content truncated to 500 words