RICOH at SemEval-2016 Task 1: IR-based Semantic Textual Similarity Estimation

Proceedings of SemEval-2016, pages 691–695, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics RICOH at SemEval-2016 Task 1: IR-based Semantic Textual Similarity Estimation Hideo Itoh RICOH Company, LTD. Institute of Information and Communication Technology 16-1, Shinei, Tsuzuki, Yokohama, Japan hideo.itoh@nts.ricoh.co.jp Abstract This paper describes our IR (Information Re- trieval) based method for SemEval 2016 task 1, Semantic Textual Similarity (STS). The main feature of our approach is to extend a conventional IR-based scheme by incorporat- ing word alignment information. This enables us to develop a more fine-grained similarity measurement. In the evaluation results, we have seen that the proposed method improves upon a conventional IR-based method on av- erage. In addition, one of our submissions achieved the best performance for the “poste- diting” data set. 1 Introduction Given two sentences, Semantic Textual Similarity (STS) measures their degree of semantic equiva- lence (Agirre et al., 2015). This fundamental func- tionality can be used for many applications such as text search, classification and clustering. This paper describes our monolingual (English) STS system which participated in SemEval 2016 task 1. Our objective is to improve a conventional IR (Information Retrieval) based method for the STS task. In general, an IR-based method estimates semantic similarity between given sentences using similarity between document search results which are obtained using each sentence as a search query. This scheme allows us to utilize a large document database for handling diverse semantic phenomena. However, in our preliminary experiments using past SemEval test data, a conventional IR-based method was not so effective. In failure analysis of the method, we found the following: • It is not sufficient only to measure the common- ality among sentences. • It is necessary to measure the importance of the identified commonality in each sentence. Based on these findings, we propose a new IR-based method for STS. In this method, word alignment techniques are applied to refine the assessment of commonality. The importance of the commonality in each sentence is measured using IR techniques. 2 Related Work Let us define a conventional IR-based approach more formally and generally. For a given text T, a document retrieval is performed using T as a search query. The search target U is a set of documents. As a result, a document set R(T, U) is obtained. Be- cause U has been fixed in our research, we denote R(T, U) as R(T). The semantic similarity between two texts T1 and T2 is measured as the similarity between document sets R(T1) and R(T2). Explicit Semantic Analysis (ESA) (Gabrilovich and Markovitch, 2007) is a well-known IR-based ap- proach. In their research, Wikipedia articles are used as U. R(T) is regarded as a vector of documents and the cosine similarity of vectors is used to measure the semantic similarity. In previous STS competitions, we can find IR- based approaches in Buscaldi et al. (2013), more re- cently Buscaldi et al. (2015). In their research, R(T) is regarded as a list of ranked documents with search scores. They defined …

Content truncated to 500 words