Comparison of Algorithmic and Human Assessments of Sentence Similarity

International Joint Conference on Natural Language Processing, pages 1306–1311, Nagoya, Japan, 14-18 October 2013. Comparison of Algorithmic and Human Assessments of Sentence Similarity John G. Mersch Department of Mathematics Xavier University of Louisiana New Orleans, LA 70125 USA jmersch@xula.edu R. Raymond Lang Department of Computer Science Xavier University of Louisiana New Orleans, LA 70125 USA rlang@xula.edu Abstract This paper describes a new method, based on information theory, for measuring sen- tence similarity. The method first com- putes the information content (IC) of de- pendency triples using corpus statistics generated by processing the Open Amer- ican National Corpus (OANC) with the Stanford Parser. We define the similarity of two sentences as a function of (1) the similarity of their constituent dependency triples, and (2) the position of the triples in their respective dependency trees. We ap- ply the algorithm to 15 pairs of sentences that were also given to human subjects to assign a similarity score. The human- and computer-generated scores are compared; the results are promising, but point to the need for further refinement. 1 Introduction This project seeks to develop an algorithm that measures the extent to which the meanings of two given sentences overlap. Our plan is to use such an algorithm in a clustering application (Lang and Mersch, 2012). The technique described in this paper extends previous work applying an information-theoretic definition of similarity to a number of different domains (Lin, 1998). Lin’s information-theoretic definition of similarity performs as well as or bet- ter than other information-theoretic similarity met- rics that leverage domain specifics (Resnik, 1995; Wu and Palmer, 1994). The metric being proposed in this paper shares characteristics of word co-occurrence methods and descriptive feature-based methods (Li et al., 2006), in addition to using structural information provided by the Stanford Parser (Klein and Man- ning, 2003). We test this metric on 15 pairs of sen- tences, each of which was assessed for similarity by 40 fluent English speakers. 2 Background & Related Work Methods that detect similarity of long documents often utilize co-occurring words (Salton, 1988), since similar texts share a high number of words. But this does not transfer well to short, sentence- length texts, since language allows similar mean- ings to be expressed using different vocabularies. Existing text similarity measures suffer from drawbacks. Vector-based methods employ high- dimensional, sparse representations that are com- putationally inefficient (Landauer et al., 1998; Salton, 1988; Burgess et al., 1998). Some meth- ods rely on extensive manual preprocessing (Mc- Clelland and Kawamoto, 1986), making them im- practical for large-scale use. Still other methods suffer from domain dependency (Li et al., 2006). Related work on text similarity may be grouped into three categories: 1. Methods based on word co-occurrence (i.e. “bag of words” methods) disregard the im- pact of word order on meaning (Meadow et al., 1999); thus, the two sentences: T1: The cat killed the mouse. T2: The mouse killed the cat. are regarded as identical, since they use the same words. Documents are represented as vectors in an …

Content truncated to 500 words