AZMAT: Sentence Similarity Using Associative Matrices

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 159–163, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics Azmat: Sentence Similarity using Associative Matrices Evan Jaffe Lifeng Jin David King Department of LinguisticsMarten van Schijndel The Ohio State University {jaffe.59, jin.544, king.2138}@osu.edu, vanschm@ling.osu.edu Abstract This work uses recursive autoencoders (Socher et al., 2011), word embeddings (Pennington et al., 2014), associative matrices (Schuler, 2014) and lexical overlap features to model human judgments of sentential similarity on SemEval-2015 Task 2: English STS (Agirre et al., 2015). Results show a modest positive correlation between system predictions and human similarity scores, ranking 69th out of 74 submitted systems. 1 Introduction This work uses a support vector machine (SVM) to determine the similarity of sentence pairs, tak- ing as input the similarity judgments of four subsys- tems: a set of surface features, unfolding recursive autoencoders (URAE; Socher et al., 2011), Global Vector word embeddings (GloVe; Pennington et al., 2014), and the Schuler (2014) associative matrix ap- proach using the Nguyen et al. (2012) Generalized Categorial Grammar (GCG). Evaluation is run on SemEval 2015 task 2, Semantic Textual Similarity (STS), which includes a corpus of human similarity judgments. The test set consists of 3000 randomly chosen sentence pairs from a corpus of 8500 pairs, which spans five domains (news headlines, image captions, student answers, forum responses, and sen- tences about belief). Similarity scores range from 0 (no similarity) to 5 (complete semantic equivalence). 2 System Overview All subsystems in Azmat are trained with sentences from previous SemEval tasks 2012 - 2014 (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014). In total, 15,406 sentences were selected from the Microsoft video, news headlines, images, and para- phrase datasets. The main purpose of the subsys- tems (excluding surface features) is to generate bina- rized phrase-structure trees, which are used to create cosine similarity features between multiple levels of paired sentences. The URAE subsystem prepro- cesses training sentences by parsing them with the Stanford Parser (Klein and Manning, 2003) and then binarizing. The associative matrix and GloVe sub- systems use GCG parses of the training sentences, obtained by training the Berkeley parser (Petrov and Klein, 2007) with the Nguyen et al. (2012) GCG re- annotated Penn Treebank. GCG parse trees are con- verted into typed dependency graphs and binarized. Around 2% of the sentences fail to parse; these are omitted from the training set. 2.1 Subsystem Combination Because vector composition methods vary across subsystems, this work incorporates multiple subsys- tems to give insight on which composition meth- ods perform better at finding semantic textual sim- ilarity. For each sentence, each subsystem generates a single binarized phrase-structure tree with a sin- gle embedding labeled at each node. Cosine similar- ity scores are calculated between each node in one tree and each node in the other tree, allowing com- parison between input sentences at and across leaf, phrasal and sentential levels. These similarity scores are used to generate a feature vector for training …

Content truncated to 500 words