Duluth : Measuring Cross-Level Semantic Similarity with First and Second Order Dictionary Overlaps

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 247–251, Dublin, Ireland, August 23-24, 2014. Duluth : Measuring Cross–Level Semantic Similarity with First and Second–Order Dictionary Overlaps Ted Pedersen Department of Computer Science University of Minnesota Duluth, MN 55812 tpederse@d.umn.edu Abstract This paper describes the Duluth systems that participated in the Cross–Level Se- mantic Similarity task of SemEval–2014. These three systems were all unsupervised and relied on a dictionary melded together from various sources, and used first–order (Lesk) and second–order (Vector) over- laps to measure similarity. The first–order overlaps fared well according to Spear- man’s correlation (top 5) but less so rela- tive to Pearson’s. Most systems performed at comparable levels for both Spearman’s and Pearson’s measure, which suggests the Duluth approach is potentially unique among the participating systems. 1 Introduction Cross–Level Semantic Similarity (CLSS) is a novel variation on the problem of semantic simi- larity. As traditionally formulated, pairs of words, pairs of phrases, or pairs of sentences are scored for similarity. However, the CLSS shared task (Jurgens et al., 2014) included 4 subtasks where pairs of different granularity were measured for semantic similarity. These included : word-2- sense (w2s), phrase-2-word (p2w), sentence-2- phrase (s2p), and paragraph-2-sentence (g2s). In addition to different levels of granularity, these pairs included slang, jargon and other examples of non–standard English. We were drawn to this task because of our long– standing interest in semantic similarity. We have pursued approaches ranging from those that rely on structured knowledge sources like WordNet (e.g., WordNet::Similarity) (Pedersen et al., 2004) to those that use distributional information found This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ in raw text (e.g., SenseClusters) (Purandare and Pedersen, 2004). Our approach in this shared task is a bit of both, but relies on using definitions for each item in a pair so that similarity can be mea- sured using first or second–order overlaps. A first–order approach finds direct matches be- tween the words in a pair of definitions. In a second–order approach each word in a definition is replaced by a vector of the words it co–occurs with, and then the vectors for all the words in a definition are averaged together to represent the definition. Then, similarity can be measured by finding the cosine between pairs of these vectors. We decided on a definition based approach since it had the potential to normalize the differences in granularity of the pairs. The main difficulty in comparing definitions is that they can be very brief or may not even ex- ist at all. This is why we combined various dif- ferent kinds of resources to arrive at our dictio- nary. While we achieved near total coverage of words and senses, phrases were sparsely covered, and sentences and paragraphs had no coverage. In those cases we used the text of the phrase, sentence or paragraph to serve as its own definition. The …

Content truncated to 500 words