DalGTM at SemEval-2016 Task 1: Importance-Aware Compositional Approach to Short Text Similarity

Proceedings of SemEval-2016, pages 765–770, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics DalGTM at SemEval-2016 Task 1: Importance-Aware Compositional Approach to Short Text Similarity Jie Mei, Aminul Islam, Evangelos Milios Faculty of Computer Science Dalhousie University, Canada {jmei,islam,eem}@cs.dal.ca Abstract This paper describes our system submission to the SemEval 2016 English Semantic Textual Similarity (STS) shared task. The proposed system is based on the compositional text sim- ilarity model, which aggregates pairwise word similarities for computing the semantic simi- larity between texts. In addition, our system combines word importance and word similar- ity to build an importance-similarity matrix. Three different word similarity measures are used in our three submitted runs. The eval- uation results show that taking into account context dependent word importance informa- tion improves performance. However, the per- formance of the system varies drastically be- tween different evaluation subsets. The best of our submitted runs achieves rank 60th with weighted mean Pearson correlation to human judgements of 0.6892. 1 Introduction Semantic Textual Similarity (STS) measures the de- gree of equivalence in the underlying semantics of paired natural language texts. It is an extensively researched problem with applications widely used in many research areas including natural language processing, information retrieval, and text mining. The STS task has been held annually since 2012 (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirrea et al., 2015) to encourage research into understanding sentence-level semantics. Sys- tems for this task compute semantic similarity scores for paired text snippets. Performance is evaluated by Figure 1: The general procedure of the compositional text similarity measures. They take three general steps: tok- enize the input text, compute pairwise word similarities be- tween all words, and aggregate the resulting scores to a sen- tence level textual similarity score. {w11, w12, . . . , w1n} and {w21, w22, . . . , w1m} are the tokenized words from Text 1 and Text 2, respectively. Each node in the middle represents a vector of pairwise similarity values computed by one word from Text 1 and all distinct words from Text 2. the Pearson correlation between the system scores and human judgements. This paper describes our system submission to the SemEval 2016 STS shared task (Agirre et al., 2016). The proposed system is based on the compositional text similarity model, which have been broadly re- searched in the literature by (Mihalcea et al., 2006; Li et al., 2006; Islam and Inkpen, 2006; Ho et al., 2010; Islam et al., 2012; B¨ar, 2013). The composi- tional text similarity model makes use of word-level 765 Figure 2: The procedure of the proposed importance-aware similarity measure. The general compositional procedure in Fig. 1 uses context independent similarity value. In addition, this measure takes into account context dependent word importance information. It first computes the word importance value w′ using Eq. 3 and adapted into every entry sim′ in the importance-similarity matrix using Eq. 4. similarity values as the building blocks to compute sentence-level semantics. Computing …

Content truncated to 500 words