BIT at SemEval-2016 Task 1: Sentence Similarity Based on Alignments and Vector with the Weight of Information Content

Proceedings of SemEval-2016, pages 686–690, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics BIT at SemEval-2016 Task 1: Sentence Similarity Based on Alignments and Vector with the Weight of Information Content Hao Wu 1, Heyan Huang∗1, Wenpeng Lu 2 1Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, School of Computer Science, Beijing Institute of Technology, Beijing, 100081, China 2QiLu University of Technology wuhao123@bit.edu.cn, hhy63@bit.edu.cn, lwp@qlu.edu.cn Abstract This paper describes three unsupervised sys- tems for determining the semantic similarity between two short texts or sentences submit- ted to the SemEval 2016 Task 1, all of which make use of only off-the-shelf software and data making them easy to replicate. Two systems achieved a similar Pearson correlation coefficient (0.64661 by simple vector, 0.65319 by word alignments). We include experiments on using our alignment based system on evalu- ation data from the 2014 and 2015 STS shared task. The results suggest that beyond the core similarity algorithm, other factors such as data preprocessing and use of domain-specific knowledge are also important to similarity prediction performance. 1 Introduction Given two short texts or sentences, similarity sys- tems or models should output a score that reflects how similar the two texts are in meaning. Semantic textual similarity (STS) formalizes an operation that is an important component of many natural language processing systems and has generated substantial interest within the research community (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015). STS methods can be applied in example-based machine translation, machine trans- lation evaluation, information retrieval, text summa- rization, question answering, and recommendation systems. ∗Corresponding author 2 System Overview In STS 2016, we submitted three system runs, and all of which were unsupervised. They could be generally divided into two kinds: vector based and alignment based. 2.1 Run 1: Simple Vector Method In this run, we use a sentence vector derived from word embeddings obtained from word2vec (Mikolov et al., 2013). Using these sentence level vector representations, the similarity between two texts can be computed using the cosine operation. We train word embeddings by running the word2vec toolkit1 over the fifth edition of the Gigaword corpus (LDC2011T07). We preprocess the Gigaword data with the following tools from the Moses machine translation toolkit (Koehn et al., 2007): the data is tokenized using tokenizer.perl; truecase.perl4 is used to standardize capitalizing. As illustrated in Equation (1), we construct the sentence vector ⃗s by simply summing together the word embeddings, ⃗ti, associated with each token in a sentence. ⇀s = |s| � i=1 ⃗ti . (1) Here |s| is the number of tokens that the sentence contains. The similarity between a pair of sentences is computed as the cosine of their associated sentence level embedding vectors. 686 2.2 Run 2: Weighted Vector Method The above method weights all word embeddings e- qually. We submitted an alternative run that weights the word embeddings by the information content (IC) of the concepts referenced by their word …

Content truncated to 500 words