BIT at SemEval-2017 Task 1: Using Semantic Information Space to Evaluate Semantic Textual Similarity

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 77–84, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics BIT at SemEval-2017 Task 1: Using Semantic Information Space to Evaluate Semantic Textual Similarity Hao Wu, Heyan Huang∗, Ping Jian, Yuhang Guo, Chao Su Beijing Engineering Research Center of High Volume Language Information Processing and Cloud Computing Applications, School of Computer Science, Beijing Institute of Technology, Beijing, China {wuhao123, hhy63, pjian, guoyuhang, suchao}@bit.edu.cn Abstract This paper presents three systems for semantic textual similarity (STS) evaluation at SemEval-2017 STS task. One is an unsupervised system and the other two are supervised systems which simply employ the unsupervised one. All our systems mainly depend on the semantic information space (SIS), which is constructed based on the semantic hierarchical taxonomy in WordNet, to compute non-overlapping content (IC) of sentences.information Our team ranked 2nd among 31 participating teams by the primary score of Pearson correlation coefficient (PCC) mean of 7 tracks and achieved the best performance on Track 1 (AR-AR) dataset. 1 Introduction Given two snippets of text, semantic textual simi- larity (STS) measures the degree of equivalence in the underlying semantics. STS is a basic but im- portant issue with multitude of application areas in natural language processing (NLP) such as exam- ple based machine translation (EBMT), machine translation evaluation, information retrieval (IR), question answering (QA), text summarization and so on. The SemEval STS task has become the most famous activity for STS evaluation in recent years and the STS shared task has been held annual- ly since 2012 (Agirre et al., 2012, 2013, 2014, 2015, 2016; Cer et al., 2017), as part of the SemEval/*SEM family of workshops. The orga- nizers have set up publicly available datasets of sentence pairs with similarity scores from human annotators, which are up to more than 16,000 ∗Corresponding author sentence pairs for training and evaluation, and attracted a large number of teams with a variety of systems to participate the competitions. Generally, STS systems could be divided into two categories: One kind is unsupervised sys- tems (Li et al., 2006; Mihalcea et al., 2006; Is- lam and Inkpen, 2008; Han et al., 2013; Sultan et al., 2014b; Wu and Huang, 2016), some of which are appeared for a long time when there wasn’t enough training data; The other kind is supervised systems (B¨ar et al., 2012; ˇSari´c et al., 2012; Sultan et al., 2015; Rychalska et al., 2016; Brychc´ın and Svoboda, 2016) applying machine learning algorithms, including deep learning, after adequate training data has been constructed. Each kind of methods has its advantages and application areas. In this paper, we present three systems, one unsupervised system and two supervised systems which simply make use of the unsupervised one. 2 Preliminaries Following the standard argumentation of informa- tion theory, Resnik (1995) proposed the definition of the information content (IC) of a concept as follows: IC (c) = − log P(c), (1) where P(c) refers to statistical frequency of con- cept c. Since information …

Content truncated to 500 words