QLUT at SemEval-2017 Task 1: Semantic Textual Similarity Based on Word Embeddings

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 150–153, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics QLUT at SemEval-2017 Task 1: Semantic Textual Similarity Based on Word Embeddings Fanqing Meng, Wenpeng Lu*, Yuteng Zhang, Jinyong Cheng, Yuehan Du, Shuwang Han Institute of Intelligent Information Processing, School of Information QiLu University of Technology, Jinan, Shandong, China mengfanqing678@163.com, lwp@qlu.edu.cn, zhangyuteng1029@163.com, cjy@qlu.edu.cn, amaris_du@163.com, hanshuwang0909@163.com Abstract This paper reports the details of our sub- missions in the task 1 of SemEval 2017. This task aims at assessing the semantic textual similarity of two sentences or texts. We submit three unsupervised systems based on word embeddings. The differ- ences between these runs are the various preprocessing on evaluation data. The best performance of these systems on the eval- uation of Pearson correlation is 0.6887. Unsurprisingly, results of our runs demon- strate that data preprocessing, such as to- kenization, lemmatization, extraction of content words and removing stop words, is helpful and plays a significant role in im- proving the performance of models. 1 Introduction Semantic Textual Similarity (STS) has been held in SemEval since 2012 (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015; Agirre et al., 2016), which is a basic task in natural language processing (NLP) field. It aims at computing the semantic similarity of two short texts or sentences, and the result will be evaluated on a gold standard set, which is made by several official annotators (Cer et al., 2017). In recent years, as an unsupervised method, word embed- ding (Mikolov et al., 2013a) becomes more and more popular in SemEval (Jimenez, 2016; Wu et al., 2016). The paper describes the submission of our sys- tems to STS 2017, which utilize word embedding method. Different from some teams who have _______________________ *Corresponding author Sentence 1 Sentence 2 Tokenization Extraction of content words Lemmatization Wikipedia Word embeddings Sentence similarity Normalization Run1 Run2 Run3 1 2 3 Figure 1: Framework of system. used word embedding described above, what we pay attention to is the point of preprocessing eval- uation data. With this consideration, we process the evaluation data with different method in order to verify whether it works or not. The framework of our systems is showed in Figure 1. Its simple description is as follows: Tokenization: This is to tokenize the two sen- tences of the system’s input. Though the English sentence is tokenized naturally, the punctuations are not. For instance, the sentence “A person is on a baseball team.” will be tokenized to “A person is on a baseball team .”. Extraction of content words: In this process, content words of the tokenized sentence will be extracted. For example, the tokenized sentence 150 “A person is on a baseball team .” turns into “per- son is baseball team”. In this paper, content words include nouns, verbs, adverbs or adjectives. Lemmatization: It is known that words in English sentences have a variety of forms. This operation will lemmatize …

Content truncated to 500 words