Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity

Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 87–96, Osaka, Japan, December 11-17 2016. Task-Oriented Intrinsic Evaluation of Semantic Textual Similarity Nils Reimers†, Philip Beyer‡, Iryna Gurevych†§ †Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universit¨at Darmstadt ‡ zeb.rolfes.schierenbeck.associates GmbH § Ubiquitous Knowledge Processing Lab (UKP-DIPF) German Institute for Educational Research http://www.ukp.tu-darmstadt.de Abstract Semantic Textual Similarity (STS) is a foundational NLP task and can be used in a wide range of tasks. To determine the STS of two texts, hundreds of different STS systems exist, however, for an NLP system designer, it is hard to decide which system is the best one. To answer this question, an intrinsic evaluation of the STS systems is conducted by comparing the output of the system to human judgments on semantic similarity. The comparison is usually done using Pearson correlation. In this work, we show that relying on intrinsic evaluations with Pearson cor- relation can be misleading. In three common STS based tasks we could observe that the Pearson correlation was especially ill-suited to detect the best STS system for the task and other evalu- ation measures were much better suited. In this work we define how the validity of an intrinsic evaluation can be assessed and compare different intrinsic evaluation methods. Understanding of the properties of the targeted task is crucial and we propose a framework for conducting the intrinsic evaluation which takes the properties of the targeted task into account. 1 Introduction Semantic Textual Similarity (STS) is the foundational NLP task of determining the degree of semantic similarity between two texts. Most STS systems compute the similarity score between two texts on a fixed scale, for example a scale between 0 and 5, with 0 indicating the semantics are completely independent and 5 indicating semantic equivalence. In recent years, the number and quality of systems that rate the STS between texts have increased, as has the number of tasks where such systems are used. Textual similarity is an active research field and was part of several shared tasks. In 2012, the pilot Semantic Textual Similarity (STS) Task (Agirre et al., 2012) was established at the Semantic Evaluation (SemEval) workshop. Further shared tasks on text similarity were part of SemEval 2013 (Agirre et al., 2013), SemEval 2014 (Agirre et al., 2014), SemEval 2015 (Agirre et al., 2015), and SemEval 2016 (Agirre et al., 2016). For the latest shared task on semantic textual similarity at SemEval 2016, 43 teams were submitting 119 different systems, depicting the large interest in this field. STS is a foundational NLP technique, however, STS systems are seldom used for the sole purpose of measuring the similarity of two texts. Often they are used in a larger context. Examples for such tasks can be found in the field of Automatic Essay Grading (Attali et al., 2006), Plagiarism Detection (Potthast et al., 2012), Automated Text Summarization (Barzilay and Elhadad, 1997), Question Answering (Lin and Pantel, 2001), or Link Discovery (He, 2009). In this …

Content truncated to 500 words