WOLVESAAR at SemEval-2016 Task 1: Replicating the Success of Monolingual Word Alignment and Neural Embeddings for Semantic Textual Similarity

Proceedings of SemEval-2016, pages 634–639, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics WOLVESAAR at SemEval-2016 Task 1: Replicating the Success of Monolingual Word Alignment and Neural Embeddings for Semantic Textual Similarity Hanna Bechara, Rohit Gupta, Liling Tan, Constantin Or˘asan, Ruslan Mitkov, Josef van Genabith University of Wolverhampton / UK, Universit¨at des Saarlandes / Germany Deutsches Forschungszentrum f¨ur K¨unstliche Intelligenz / Germany {hanna.bechara, r.gupta, corsasan, r.mitkov}@wlv.ac.uk, liling.tan@uni-saarland.de, josef.van genabith@dfki.de Abstract This paper describes the WOLVESAAR sys- tems that participated in the English Seman- tic Textual Similarity (STS) task in SemEval- 2016. We replicated the top systems from the last two editions of the STS task and ex- tended the model using GloVe word embed- dings and dense vector space LSTM based sentence representations. We compared the difference in performance of the replicated system and the extended variants. Our vari- ants to the replicated system show improved correlation scores and all of our submissions outperform the median scores from all partic- ipating systems. 1 Introduction Semantic Textual Similarity (STS) is the task of as- signing a real number score to quantify the semantic likeness of two text snippets. Similarity measures play a crucial role in various areas of text processing and translation technologies ranging from improv- ing information retrieval rankings (Lin and Hovy, 2003; Corley and Mihalcea, 2005) and text summa- rization to machine translation evaluation and en- hancing matches in translation memory and termi- nologies (Resnik and others, 1999; Ma et al., 2011; Banchs et al., 2015; Vela and Tan, 2015). The annual SemEval STS task (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015) provides a platform where systems are evaluated on the same data and evaluation criteria. 2 DLS System from STS 2014 and 2015 For the past two editions of the STS task, the top per- forming submissions are from the DLS@CU team (Sultan et al., 2014b; Sultan et al., 2015). Their STS2014 submission is based on the pro- portion of overlapping content words between the two sentences treating semantic similarity as a monotonically increasing function of the degree to which two sentences contain semantically similar units and these units occur in similar semantic con- texts (Sultan et al., 2014b). Essentially, their seman- tic metric is based on the proportion of aligned con- tent words between two sentences, formally defined as: prop(1) Al = |{i : [∃j : (i, j) ∈ Al] and w(1) i ∈ C}| |{i : w(1) i ∈ C}| (1) where prop(1) Al is the monotonic proportion of the semantic unit alignment from a set of alignments Al that maps the positions of the words (i, j) between sentences S(1) and S(2), given that the aligned units belong to a set of content words, C. Since the pro- portion is monotonic, the equation above only pro- vides the proportion of semantic unit alignments for S(1). The Al alignments pairs are automatically an- notated by a monolingual word aligner (Sultan et al., 2014a) that …

Content truncated to 500 words