SemantiKLUE: Semantic Textual Similarity with Maximum Weight Matching

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 111–116, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemantiKLUE: Semantic Textual Similarity with Maximum Weight Matching Nataliia Plotnikova and Gabriella Lapesa and Thomas Proisl and Stefan Evert Friedrich-Alexander-Universit¨at Erlangen-N¨urnberg (FAU) Professur f¨ur Korpuslinguistik Bismarckstr. 6, 91054 Erlangen, Germany {nataliia.plotnikova,gabriella.lapesa,thomas.proisl,stefan.evert}@fau.de Abstract This paper describes the SemantiKLUE sys- tem (Proisl et al., 2014) used for the SemEval- 2015 shared task on Semantic Textual Simi- larity (STS) for English. The system was de- veloped for SemEval-2013 and extended for SemEval-2014, where it participated in three tasks and ranked 13th out of 38 submissions for the English STS task. While this year’s submission ranks 46th out of 73, further ex- periments on the selection of training data led to notable improvements showing that the system could have achieved rank 22 out of 73. We report a detailed analysis of those training selection experiments in which we tested different combinations of all the avail- able STS datasets, as well as results of a qual- itative analysis conducted on a sample of the sentence pairs for which SemantiKLUE gave wrong STS predictions. 1 Introduction The SemEval-2015 task on ”Semantic Textual Sim- ilarity for English” (Agirre et al., 2015) is a rerun of the corresponding task from SemEval-2014 with new test data and updated categories. The predic- tions of participating systems were evaluated against manually annotated and subsequently filtered data. STS was measured on a scale ranging from 0 (no similarity at all) to 5 (total equivalence). SemantiK- LUE, developed in 2014, uses a distributional bag- of-words model as well as a word-to-word align- ment for each pair of sentences based on a maximum weight matching algorithm. Our SemEval-2015 submission for all 5 test cat- egories (headlines, images, belief, answers-forums, answers-students) was based on the training data set from 2014 with 2234 sentence pairs from 3 cate- gories, namely paraphrase sentence pairs (MSRpar), sentence pairs from video descriptions (MSRvid) and MT evaluation sentence pairs (SMTeuroparl). Follow up experiments conducted after the submis- sion deadline showed us that this training config- uration was far from optimal, and that our system would have benefited a lot from a better training, as we managed to significantly improve the overall scores. With the best training configuration, Seman- tiKLUE would have ranked 22nd out of 73 submis- sions (11th out of 28 teams), with a weighted mean of Pearson correlation coefficients over all test cate- gories of 0.7508 (best system: 0.8015) In the following sections, we first give a short overview of the system (Section 2), and then we de- scribe the follow-up experiments that allowed us to define the best training data set in terms of its sub- sets (Section 3); finally, we present the results of a qualitative analysis of the performance of our sys- tem (Section 4). 2 System Description SemantiKLUE combines supervised and unsuper- vised approaches for the computation of textual sim- ilarity: a number of similarity measures are com- puted and passed …

Content truncated to 500 words