DT_Team at SemEval-2017 Task 1: Semantic Similarity Using Alignments, Sentence-Level Embeddings and Gaussian Mixture Model Output

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 120–124, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics DT Team at SemEval-2017 Task 1: Semantic Similarity Using Alignments, Sentence-Level Embeddings and Gaussian Mixture Model Output Nabin Maharjan, Rajendra Banjade, Dipesh Gautam, Lasang J. Tamang and Vasile Rus Department of Computer Science / Institute for Intelligent Systems The University of Memphis Memphis, TN, USA {nmharjan,rbanjade,dgautam,ljtamang,vrus}@memphis.edu Abstract We describe our system (DT Team) sub- mitted at SemEval-2017 Task 1, Seman- tic Textual Similarity (STS) challenge for English (Track 5). We developed three dif- ferent models with various features includ- ing similarity scores calculated using word and chunk alignments, word/sentence em- beddings, and Gaussian Mixture Model (GMM). The correlation between our sys- tem’s output and the human judgments were up to 0.8536, which is more than 10% above baseline, and almost as good as the best performing system which was at 0.8547 correlation (the difference is just about 0.1%). Also, our system produced leading results when evaluated with a sep- arate STS benchmark dataset. The word alignment and sentence embeddings based features were found to be very effective. 1 Introduction Measuring the Semantic Textual Similarity (STS) is to quantify the semantic equivalence between given pair of texts (Banjade et al., 2015; Agirre et al., 2015). For example, a similarity score of 0 means that the texts are not similar at all while a score of 5 means that they have same meaning. In this paper, we describe our system DT Team and the three different runs that we submitted to this year’s SemEval shared task on STS English track (Track 5; Agirre et al. (2017)). We ap- plied Support Vector Regression (SVR), Linear Regression (LR) and Gradient Boosting Regressor (GBR) with various features (see § 3.4) in order to predict the semantic similarity of texts in a given pair. We also report the results of our models when evaluated with a separate STS benchmark dataset created recently by the STS task organizers. 2 Preprocessing The preprocessing step involved tokenization, lemmatization, POS-tagging, name-entity recog- nition and normalization (e.g. pc, pct, % are nor- malized to pc). The preprocessing steps were same as our DTSim system (Banjade et al., 2016). 3 Feature Generation We generated various features including similar- ity scores generated using different methods. We describe next the word-to-word and sentence-to- sentence similarity methods used in our system. 3.1 Word-to-Word Similarity We used the word2vec (Mikolov et al., 2013)1 vectorial word representation, PPDB database (Pavlick et al., 2015)2, and WordNet (Miller, 1995) to compute similarity between words. Please see DTSim system description (Banjade et al., 2016) for additional details. 3.2 Sentence-to-Sentence Similarity 3.2.1 Word Alignment Method We lemmatized all content words and aligned them optimally using the Hungarian algorithm (Kuhn, 1955) implemented in the SEMILAR Toolkit (Rus et al., 2013). The process is the same as finding the maximum weight matching in a weighted bi-partite graph. The nodes are words and the weights are the similarity scores between …

Content truncated to 500 words