SemEval-2016 Task 2: Interpretable Semantic Textual Similarity

Proceedings of SemEval-2016, pages 512–524, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SemEval-2016 Task 2: Interpretable Semantic Textual Similarity Eneko Agirre∗, Aitor Gonzalez-Agirre, I˜nigo Lopez-Gazpio Montse Maritxalar, German Rigau and Larraitz Uria IXA NLP group, University of the Basque Country Manuel Lardizabal 1, 20.018 Donostia, Basque Country montse.maritxalar@ehu.eus Abstract The final goal of Interpretable Semantic Tex- tual Similarity (iSTS) is to build systems that explain which are the differences and com- monalities between two sentences. The task adds an explanatory level on top of STS, for- malized as an alignment between the chunks in the two input sentences, indicating the re- lation and similarity score of each alignment. The task provides train and test data on three datasets: news headlines, image captions and student answers. It attracted nine teams, total- ing 20 runs. All datasets and the annotation guideline are freely available1 1 Introduction Semantic Textual Similarity (STS) (Agirre et al., 2015) measures the degree of equivalence in the un- derlying semantics of paired snippets of text. The idea of Interpretable STS (iSTS) is to explain why two sentences may be related/unrelated, by supple- menting the STS similarity score with an explana- tory layer. Our final goal would be to enable interpretable systems, that is, systems that are able to explain which are the differences and commonalities be- tween two sentences. For instance, let’s assume the following two sentences drawn from a corpus of news headlines: 12 killed in bus accident in Pakistan 10 killed in road accident in NW Pakistan ∗* Authors listed in alphabetical order 1http://at.qrci.org/semeval2016/task2/ The output of such a system would be something like the following: The two sentences talk about accidents with casualties in Pakistan, but they dif- fer in the number of people killed (12 vs. 10) and level of detail: the first one speci- fies that it is a bus accident, and the second one specifies that the location is NW Pak- istan. While giving such explanations comes naturally to people, constructing algorithms and computa- tional models that mimic human level performance represents a difficult Natural Language Understand- ing (NLU) problem, with applications in dialogue systems, interactive systems and educational sys- tems. In the iSTS 2015 pilot task (Agirre et al., 2015), we defined a first step of such an ambitious system, which we follow in 2016. Given the input (a pair of sentences), participant systems need first to identify the chunks in each sentence, and then, align chunks across the two sentences, indicating the relation and similarity score of each alignment. The relation can be one of equivalence, opposition, specificity, sim- ilarity or relatedness, and the similarity score can range from 1 to 5. Unrelated chunks are left un- aligned. An optional tag can be added to alignments for the cases where there is a difference in factuality or polarity. See Figure 1 for the manual alignment of the two sample sentences. The alignments between chunks in Figure 1 can be used to produce the kind of explanations …

Content truncated to 500 words