SimiHawk at SemEval-2016 Task 1: A Deep Ensemble System for Semantic Textual Similarity

Proceedings of SemEval-2016, pages 741–748, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SimiHawk at SemEval-2016 Task 1: A Deep Ensemble System for Semantic Textual Similarity Peter Potash∗, William Boag∗, Alexey Romanov∗, Vasili Ramanishka, Anna Rumshisky Dept. of Computer Science University of Massachusetts Lowell 198 Riverside St, Lowell, MA 01854 {ppotash,wboag,aromanov,vramanis,arum}@cs.uml.edu Abstract This paper describes the SimiHawk system submission from UMass Lowell for the core Semantic Textual Similarity task at SemEval- 2016. We built four systems: a small feature- based system that leverages word alignment and machine translation quality evaluation metrics, two end-to-end LSTM-based sys- tems, and an ensemble system. The LSTM- based systems used either a simple LSTM architecture or a Tree-LSTM structure. We found that of the three base systems, the feature-based model obtained the best results, outperforming each LSTM-based model’s correlation by .06. Ultimately, the ensemble system was able to outperform the base sys- tems substantially, obtaining a weighted Pear- son correlation of 0.738, and placing 7th out of 115 participating systems. We find that the en- semble system’s success comes largely from its ability to form a consensus and eliminate complementary noise from its base systems’ predictions. 1 Introduction The task of semantic textual similarity (STS) has been developed over the past several SemEval com- petitions with the idea of capturing the degree of similarity in the meaning conveyed by two snippets of text (usually two sentences). In that respect, the STS task can be seen as similar to such tasks as paraphrase detection (Xu et al., 2015), recognizing textual entailment (RTE) (Negri et al., 2012), and semantic relatedness (Marelli et al., 2014a). The ∗These three authors contributed equally to this work. STS task captures different gradations of similar- ity, rather than a binary decision, and it is symmet- ric, rather than directed, as is the case with RTE (Agirre et al., 2012). It also aims to capture a more general notion of semantic similarity that does not focus solely on the semantic relatedness derived through compositional processes. In this paper, we describe the SimiHawk system submission for the core Semantic Textual Similarity (STS) task at the SemEval-2016 competition. The STS task series (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirrea et al., 2015) has aggregated a sizable dataset of sentence pairs annotated with numeric similarity scores. The pres- ence of this dataset allows for a shift from earlier work that mostly used unsupervised learning (Cor- ley and Mihalcea, 2005; Mihalcea et al., 2006; Li et al., 2006), to the supervised approaches that lever- age the labeled data (Sultan et al., 2015; Han et al., 2015; H¨anig et al., 2015). The availability of labeled data from different genres also allows for a clearer evaluation and a better comparison across different approaches. Specifically, the task of semantic similarity is de- fined as follows: given an input of two sentences, output a real number in the range [0,5] where 0 means lowest and 5 means highest similarity. The top-performing system …

Content truncated to 500 words