UNBNLP at SemEval-2016 Task 1: Semantic Textual Similarity: A Unified Framework for Semantic Processing and Evaluation

Proceedings of SemEval-2016, pages 732–735, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics UNBNLP at SemEval-2016 Task 1: Semantic Textual Similarity: A Unified Framework for Semantic Processing and Evaluation Milton King and Waseem Gharbieh and SoHyun Park and Paul Cook Faculty of Computer Science, University of New Brunswick Fredericton, NB E3B 5A3, Canada {milton.king,waseem.gharbieh,sohyun.park,paul.cook}@unb.ca Abstract In this paper we consider several approaches to predicting semantic textual similarity us- ing word embeddings, as well as methods for forming embeddings for larger units of text. We compare these methods to several base- lines, and find that none of them outperform the baselines. We then consider both a super- vised and unsupervised approach to combin- ing these methods which achieve modest im- provements over the baselines. 1 Introduction Word embeddings (Mikolov et al., 2013) have re- cently led to improvements in a wide range of tasks in natural language processing. A number of approaches to forming embeddings for sentences, paragraphs, and documents have also recently been proposed (e.g., Le and Mikolov, 2014; Kiros et al., 2015). These methods seem particularly well suited to the task of predicting semantic textual similarity (STS), and indeed have been shown to work very well on similar tasks (Kiros et al., 2015). This paper describes the system of UNBNLP at SemEval-2016 Task 1. We first implement sev- eral baseline approaches to STS based on cosine similarity of count-based vectors representing sen- tences, with a variety of approaches to term weight- ing. We then consider approaches drawing off of word2vec (Mikolov et al., 2013), paragraph vectors (Le and Mikolov, 2014), and skip-thoughts (Kiros et al., 2015). We find that none of these approaches improve over any of our baselines. We then consider combining information from these individual methods to measuring STS. We consider an unsupervised approach based on the av- erage of the predicted similarities for a number of these individual approaches. We further consider a supervised approach in which we train ridge regres- sion with features corresponding to the similarities from these individual methods. Each of these meth- ods for combining information achieves modest im- provements over the baselines. 2 Measuring short text similarity We present several methods for measuring STS in Section 2.1. We then present approaches to combin- ing these methods in Section 2.2. 2.1 Individual methods 2.1.1 Baselines We present three baseline methods. In all of these baselines, each sentence in a pair of sentences is rep- resented as a vector, where each dimension corre- sponds to a word type (i.e., a word form). In the first approach, referred to as BASELINE- BIN, the dimensions hold binary values indicating whether the corresponding type occurs in the sen- tence. In the second approach, BASELINE-FREQ, the dimensions hold the frequency of the correspond- ing type in the sentence. For the third approach, BASELINE-TF-IDF, each dimension holds the tf-idf weight for the corresponding type in the sentence. Idf values were calculated over a 2015 dump of En- glish Wikipedia from 1 September 2015, which was …

Content truncated to 500 words