USFD at SemEval-2016 Task 1: Putting different State-of-the-Arts into a Box

Proceedings of SemEval-2016, pages 609–613, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics USFD at SemEval-2016 Task 1: Putting different State-of-the-Arts into a Box Ahmet Aker, Frederic Blain, Andres Duque∗, Marina Fomicheva+, Jurica Seva, Kashif Shah and Daniel Beck University of Sheffield, UK University of Madrid, Spain∗ Universitat Pompeu Fabra, Spain+ ahmet.aker@ f.blain@ j.seva@ kashif.shah@ sheffield.ac.uk andres.duq.fer@gmail.com, marina.fomicheva@upf.edu debeck1@sheffield.ac.uk Abstract In this paper we describe our participa- tion in the STS Core subtask which is the determination of the monolingual seman- tic similarity between pair of sentences. In our participation we adapted state-of- the-art approaches from related work ap- plied on previous STS Core subtasks and run them on the 2016 data. We inves- tigated the performance of single meth- ods but also the combination of them. Our results show that Convolutional Neu- ral Networks (CNN) are superior to both the Monolingual Word Alignment and the Word2Vec approaches. The combination of all the three methods performs slightly better than using CNN only. Our results also show that the performance of our systems varies between the datasets. 1 Introduction Semantic Textual Similarity (STS) is a metric which aims to determine the likeness between two short textual entities. Therefore, STS is widely used in many research areas such as Nat- ural Language Processing, for a large amount of tasks like Information Retrieval (IR), Natural Language Understanding (NLU) or even Ma- chine Translation (MT) evaluation, for which STS allows capturing more information than traditional metrics based on n-grams match like BLEU (Papineni et al., 2002). Part of the SemEval campaign, STS compe- tition benefits from a growing interest over the year (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirrea et al., 2015). In 2016, participants had the possibility to com- pete in two tasks: (i) STS split into ’STS Core’ and ’Cross-lingual STS’ which are respectively an English monolingual and English/Spanish bilingual subtasks; (ii) iSTS which focuses on the interpretable aspect of STS assessment. In- troduced for the first time in 2015, iSTS became a standalone task this year. This paper describes our participation in the STS Core subtask and is organised as follows: we first describe in Section 2 methods we have adapted to tackle the task. Next, we present and discuss our results (Section 3). Finally, Sec- tion 4 will conclude this system description pa- per with some remarks. 2 System description For adressing monolingual semantic similarity between two sentences different methods have been proposed in related work. In our mono- lingual STS participation we have focused on some methods that were reported as state-of- 609 the-art systems. Given that those methods have been reported separately a natural question that arises from this is what the performance is when those methods are combined. This is exactly what we did and what we propose in this pa- per. We adapted methods from related work that have been applied on the monolingual STS task and used their combined version on the new 2016 monolingual task. …

Content truncated to 500 words