MITRE: Seven Systems for Semantic Similarity in Tweets

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 12–17, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics MITRE: Seven Systems for Semantic Similarity in Tweets Guido Zarrella, John Henderson, Elizabeth M. Merkhofer and Laura Strickhart The MITRE Corporation 202 Burlington Road Bedford, MA 01730-1420, USA {jzarrella,jhndrsn,emerkhofer,lstrickhart}@mitre.org Abstract This paper describes MITRE’s participation in the Paraphrase and Semantic Similar- ity in Twitter task (SemEval-2015 Task 1). This effort placed first in Semantic Similar- ity and second in Paraphrase Identification with scores of Pearson’s r of 61.9%, F1 of 66.7%, and maxF1 of 72.4%. We detail the approaches we explored including mixtures of string matching metrics, alignments us- ing tweet-specific distributed word represen- tations, recurrent neural networks for model- ing similarity with those alignments, and dis- tance measurements on pooled latent semantic features. Logistic regression is used to tie the systems together into the ensembles submitted for evaluation. 1 Introduction Paraphrase identification is the task of judging if two texts express the same or very similar meaning. Au- tomatic identification of paraphrases has practical applications for a range of domains, including news summarization, information retrieval, essay grading, and evaluation of machine translation outputs. Fur- thermore, work on paraphrase detection tends to ad- vance the state of art in modeling semantics and se- mantic similarity in natural language in general. Current approaches to paraphrase detection vary widely. The Microsoft Research Paraphrase Corpus, with pairs of sentences from newswire text, serves as a benchmark for the task (Dolan et al., 2004). One top result on this dataset uses features from surface characteristics of text (Madnani et al., 2012). An- other system with comparable results models sen- tences as hierarchical compositions of distributed word embeddings (Socher et al., 2011). SemEval- 2015 Task 1 (Xu et al., 2015), with a corpus drawn from Twitter, offers an opportunity to test paraphrase systems in a domain with an expanded vocabulary and informal grammar. Our contribution builds upon the recent success of distributed representations of language (Mikolov et al., 2013a; Pennington et al., 2014). We further aim to minimize reliance on language- and domain- dependent tools. However we do not possess enough labeled paraphrase data to train a generalized model of word composition. Instead we explore models that examine low-dimensional relationships between individual pairs of aligned words, and combine the above with string similarity features that generalize well to out-of-vocabulary terms. In the remainder of this paper, we describe our high-performing system for modeling semantic sim- ilarity between two tweets. In Section 2 we describe the data, task, and evaluation. In Section 3 we dis- cuss details of systems we built to solve the semantic similarity task. We describe our experiments on dif- ferent parameterizations in Section 4. In Section 5 we present performance results for our ensembles and all subsystems, and in Section 6 we summarize our findings. 2 Task, data and evaluation Paraphrase and Semantic Similarity in Twitter was a shared task organized within SemEval-2015. The task organizers released 18,762 pairs …

Content truncated to 500 words