MayoNLP at SemEval-2016 Task 1: Semantic Textual Similarity based on Lexical Semantic Net and Deep Learning Semantic Model

Proceedings of SemEval-2016, pages 674–679, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics * indicates joint first author’s MayoNLP at SemEval-2016 Task 1: Semantic Textual Similarity based on Lexical Semantic Net and Deep Learning Semantic Model Naveed Afzal *, Yanshan Wang*, Hongfang Liu Department of Health Sciences Research Mayo Clinic, Rochester, MN {Afzal.Naveed, Wang.Yanshan, Liu.Hongfang}@mayo.edu Abstract Given two sentences, participating systems assign a semantic similarity score in the range of 0-5. We applied two different techniques for the task: one is based on lexical semantic net (corresponding to run 1) and the other is based on deep learning semantic model (corresponding to run 2). We also combined these two runs linearly (corresponding to run 3). Our results indicate that the two techniques perform comparably while the combination outperforms the individual ones on four out of five datasets, namely answer- answer, headlines, plagiarism, and question- question, and on the overall weighted mean of STS 2016 and 2015 datasets. 1 Introduction There is a growing need for an effective method to compute semantic similarity between two text snippets. Many natural language processing (NLP) applications can benefit from effective semantic textual similarity (STS) techniques such as paraphrase recognition (Dolan et al., 2004), textual summarization (Aliguliyev, 2009), automatic machine translation evaluation (Kauchak and Barzilay, 2006), tweets search (Sriram et al., 2010), and student answer assessment (Rus and Lintean, 2012; Niraula et al., 2013). The SemEval STS task series (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015) have provided a vital platform for this task by making available a huge collection of sentence pairs with manual annotation for each sentence pair. The objective of the task is to compute semantic similarity between a pair of sentences in the range [0, 5], and where 0 indicates no similarity and 5 indicates complete similarity. The approaches for computing semantic similarity between text snippets can be broadly classified into three approaches: information retrieval vector space model (e.g. Meadow, 1992; Sahami and Heilman, 2006), text alignment on the basis of semantic equivalence (e.g. Mihalcea et al., 2006) and machine learning models on the basis of lexical, syntactic and semantic features (e.g. Saric et al., 2012). In this paper, we describe two techniques for the SemEval 2016 English STS task: the first one adopts the text alignment technique on the basis of semantic equivalence by leveraging a semantic net and corpus statistics; and the second technique is a machine learning models where we utilize a deep learning semantic model. In the following, we present our techniques in detail and provide the evaluation results of our systems on STS 2016 and 2015 datasets. Conclusions and future directions are also provided. 2 System Description This section describes the data and our systems in detail. 2.1 Data There are five datasets in the SemEval 2016 English STS task: answer-answer, headlines, plagiarism, postediting and question-question collected form different sources in this year’s English STS task. Each dataset consists of sentence pairs with human-assigned …

Content truncated to 500 words