A Reflective View on Text Similarity

Proceedings of Recent Advances in Natural Language Processing, pages 515–520, Hissar, Bulgaria, 12-14 September 2011. A Reflective View on Text Similarity Daniel B¨ar, Torsten Zesch, and Iryna Gurevych Ubiquitous Knowledge Processing Lab Computer Science Department, Technische Universit¨at Darmstadt Hochschulstrasse 10, D-64289 Darmstadt, Germany www.ukp.tu-darmstadt.de Abstract While the concept of similarity is well grounded in psychology, text similarity is less well-defined. Thus, we analyze text similarity with respect to its definition and the datasets used for evaluation. We for- malize text similarity based on the geo- metric model of conceptual spaces along three dimensions inherent to texts: struc- ture, style, and content. We empirically ground these dimensions in a set of anno- tation studies, and categorize applications according to these dimensions. Further- more, we analyze the characteristics of the existing evaluation datasets, and use those datasets to assess the performance of com- mon text similarity measures. 1 Introduction Within the natural language processing (NLP) community, similarity between texts (text similar- ity, henceforth) is utilized in a wide range of tasks, e.g. automatic essay grading (Attali and Burstein, 2006) or paraphrase recognition (Tsatsaronis et al., 2010). However, text similarity is often used as an umbrella term covering quite different phe- nomena. Therefore, we formalize text similarity and analyze the datasets used for evaluation. We argue that the seemingly simple question “How similar are two texts?” cannot be answered independently from asking what properties make them similar. Goodman (1972) gives a good ex- ample regarding the baggage check at an airport: While a spectator might compare bags by shape, size, or color, the pilot only focuses on a bag’s weight, and the passenger compares them by des- tination and ownership. Similarly, texts also have certain inherent properties (dimensions, hence- forth) that need to be considered in any attempt to judge their similarity. Consider, for example, two novels by Leo Tolstoy1. A reader may readily argue that these novels are completely dissimilar due to different plots, people, or places (i.e. dis- similar content). On the other hand, another reader may argue that both texts are indeed highly simi- lar because of their stylistic similarities. Hence, text similarity is a loose notion unless we provide a certain frame of reference. Therefore, we intro- duce a formalization based on conceptual spaces (G¨ardenfors, 2000). Furthermore, we discuss the datasets used for evaluating text similarity mea- sures. We analyze the properties of each dataset by means of annotation studies and a critical view on the performance of common similarity measures. 2 Formalization In psychology, similarity is well formalized and captured in formal models such as the set- theoretic model (Tversky, 1977) or the geometric model (Widdows, 2004). In an attempt to over- come the traditionally loose definition of text sim- ilarity, we rely on a conceptual framework based on conceptual spaces (G¨ardenfors, 2000). In this model, objects are represented in a number of ge- ometric spaces. For example, potential spaces re- lated to countries are political affinity and geo- graphical proximity. In order to adapt this model to …

Content truncated to 500 words