FBK-HLT-NLP at SemEval-2016 Task 2: A Multitask, Deep Learning Approach for Interpretable Semantic Textual Similarity

Proceedings of SemEval-2016, pages 783–789, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics FBK-HLT-NLP at SemEval-2016 Task 2: A Multitask, Deep Learning Approach for Interpretable Semantic Textual Similarity Simone Magnolini Fondazione Bruno Kessler University of Brescia Brescia, Italy magnolini@fbk.eu Anna Feltracco Fondazione Bruno Kessler University of Pavia Pavia, Italy feltracco@fbk.eu Bernardo Magnini Fondazione Bruno Kessler Povo-Trento, Italy magnini@fbk.eu Abstract We present the system developed at FBK for the SemEval 2016 Shared Task 2 ”Inter- pretable Semantic Textual Similarity” as well as the results of the submitted runs. We use a single neural network classification model for predicting the alignment at chunk level, the re- lation type of the alignment and the similar- ity scores. Our best run was ranked as first in one the subtracks (i.e. raw input data, Student Answers), among 12 runs submitted, and the approach proved to be very robust across the different datasets. 1 Introduction The Semantic Textual Similarity (STS) task mea- sures the degree of equivalence between the mean- ing of two texts, usually sentences. In the Inter- pretable STS (iSTS) (Agirre et al., 2016) the sim- ilarity is calculated at chunk level, and systems are asked to provide the type of the relationship between two chunks, as an interpretation of the similarity. Given an input pair of sentences, participant sys- tems were asked to: (i) identify the chunks in each sentence; (ii) align chunks across the two sentences; (iii) indicate the relation between the aligned chunks and (iv) specify the similarity score of each align- ment. The iSTS task has already been the object of an evaluation campaign in 2015, as a subtask of the SemEval-2015 Task 2: Semantic Textual Similarity (Agirre et al., 2015). More in general, shared tasks for the identification and measurement of STS were organized in 2012 (Agirre et al., 2012), 2013 (Agirre et al., 2013) and 2014 (Agirre et al., 2014). Data provided to participants include three datasets: image captions (Images), pairs of sen- tences from news headlines (Headlines), and a question-answer dataset collected and annotated during the evaluation of the BEETLE II tutorial dialogue system (Student Answers) (Agirre et al., 2015). For each dataset, two subtracks were re- leased: the first with raw input data (SYS), the sec- ond with data split in gold standard chunks (GS). Given these input data, participants were required to identify the chunks in each sentence (for the first subtrack only), align chunks across the two sen- tences, specify the semantic relation of the align- ment - selecting one of the following: EQUI for equivalent, OPPO for opposite, SPE1 and SPE2 if chunk in sentence1 is more specific than chunk in sentence2 and vice versa, SIMI for similar mean- ings, REL for chunks that have related meanings, and NOALI for chunk has no corresponding chunk in the other sentence (Agirre et al., 2015)-, and pro- vide a similarity score for each alignment, from 5 (maximum similarity/relatedness) to 0 (no relation at all). In addition, an optional tag for alignments showing …

Content truncated to 500 words