SimCompass: Using Deep Learning Word Embeddings to Assess Cross-level Similarity

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 560–565, Dublin, Ireland, August 23-24, 2014. SimCompass: Using Deep Learning Word Embeddings to Assess Cross-level Similarity Carmen Banea, Di Chen, Rada Mihalcea∗ University of Michigan Ann Arbor, MI Claire Cardie Cornell University Ithaca, NY Janyce Wiebe University of Pittsburgh Pittsburgh, PA Abstract This article presents our team’s partici- pating system at SemEval-2014 Task 3. Using a meta-learning framework, we experiment with traditional knowledge- based metrics, as well as novel corpus- based measures based on deep learning paradigms, paired with varying degrees of context expansion. The framework en- abled us to reach the highest overall per- formance among all competing systems. 1 Introduction Semantic textual similarity is one of the key components behind a multitude of natural lan- guage processing applications, such as informa- tion retrieval (Salton and Lesk, 1971), relevance feedback and text classification (Rocchio, 1971), word sense disambiguation (Lesk, 1986; Schutze, 1998), summarization (Salton et al., 1997; Lin and Hovy, 2003), automatic evaluation of machine translation (Papineni et al., 2002), plagiarism de- tection (Nawab et al., 2011), and more. To date, semantic similarity research has pri- marily focused on comparing text snippets of simi- lar length (see the semantic textual similarity tasks organized during *Sem 2013 (Agirre et al., 2013) and SemEval 2012 (Agirre et al., 2012)). Yet, as new challenges emerge, such as augmenting a knowledge-base with textual evidence, assessing similarity across different context granularities is gaining traction. The SemEval Cross-level seman- tic similarity task is aimed at this latter scenario, and is described in more details in the task paper (Jurgens et al., 2014). ∗{carmennb,chenditc,mihalcea}@umich.edu This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ 2 Related Work Over the past years, the research community has focused on computing semantic relatedness us- ing methods that are either knowledge-based or corpus-based. Knowledge-based methods derive a measure of relatedness by utilizing lexical re- sources and ontologies such as WordNet (Miller, 1995) or Roget (Rog, 1995) to measure defi- nitional overlap, term distance within a graph- ical taxonomy, or term depth in the taxonomy as a measure of specificity. There are many knowledge-based measures that were proposed in the past, e.g., (Leacock and Chodorow, 1998; Lesk, 1986; Resnik, 1995; Jiang and Conrath, 1997; Lin, 1998; Jarmasz and Szpakowicz, 2003; Hughes and Ramage, 2007). On the other side, corpus-based measures such as Latent Semantic Analysis (LSA) (Landauer and Dumais, 1997), Explicit Semantic Analy- sis (ESA) (Gabrilovich and Markovitch, 2007), Salient Semantic Analysis (SSA) (Hassan and Mihalcea, 2011), Pointwise Mutual Informa- tion (PMI) (Church and Hanks, 1990), PMI-IR (Turney, 2001), Second Order PMI (Islam and Inkpen, 2006), Hyperspace Analogues to Lan- guage (Burgess et al., 1998) and distributional similarity (Lin, 1998) employ probabilistic ap- proaches to decode the semantics of words. They consist of unsupervised methods that utilize the contextual information and patterns observed in raw text to build semantic profiles of words. …

Content truncated to 500 words