LIPN-IIMAS at SemEval-2017 Task 1: Subword Embeddings, Attention Recurrent Neural Networks and Cross Word Alignment for Semantic Textual Similarity

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 208–212, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics LIPN-IIMAS at SemEval-2017 Task 1: Subword Embeddings, Attention Recurrent Neural Networks and Cross Word Alignment for Semantic Textual Similarity Ignacio Arroyo-Fern´andez Universidad Nacional Autonoma de Mexico, Mexico iaf@ciencias.unam.mx Ivan Meza Instituto de Investigaciones en Matematicas Aplicadas y en Sistemas Universidad Nacional Autonoma de Mexico, Mexico ivanvladimir@turing.iimas.unam.mx Abstract In this paper we report our attempt to use, on the one hand, state-of-the-art neural ap- proaches that are proposed to measure Se- mantic Textual Similarity (STS). On the other hand, we propose an unsupervised cross-word alignment approach, which is linguistically motivated. The neural ap- proaches proposed herein are divided into two main stages. The first stage deals with constructing neural word embeddings, the components of sentence embeddings. The second stage deals with constructing a se- mantic similarity function relating pairs of sentence embeddings. Unfortunately our competition results were poor in all tracks, therefore we concentrated our research to improve them for Track 5 (EN-EN). 1 Introduction Semantic Textual Similarity (STS) refers to the Natural Language Processing (NLP) task which is aimed at measuring the degree of similar- ity/dissimilarity between two text units (Agirre et al., 2012, 2016). In other words given a pair of text snippets (generally a pair of sentences) the task is to determine a real value (the semantic sim- ilarity score) in the interval between 0.0 and 5.0, which represents how much similar are the two sentences of a given pair. There are two main types of proposed systems in prior editions of the competition: supervised and unsupervised systems. While supervised sys- tems are expected to be highly reliable because of that they use human-annotated gold standards, un- supervised systems also are highly reliable by us- ing modest levels of linguistic knowledge. In this work we report results from both, unsupervised and supervised systems. Currently the STS task involves tracks of differ- ent nature, i.e. the monolingual and cross-lingual ones. In this paper we investigate the underlying properties in text which are relevant to measure se- mantic similarity, thus we focus our major efforts into the English-English Track 5. 2 Data We tested a couple of supervised systems. We pre- pared the STS monolingual English datasets from years 2012, 2013, 2015 and 2016. After discard- ing sentence pairs whose similarity score was ab- sent from the corresponding gold standard files, we obtained a dataset consisted of 10, 592 sen- tence pairs (6, 858 are already marked as training pairs and 3, 734 are already marked as test pairs). In order to obtain subword embeddings we trained the “fastText” method for 20, 50, 100, 200 and 300 dimensions by using the English Wikipedia (Bojanowski et al., 2016). We decided to take advantage of the capability of this method for inferring out-of-vocabulary words. This ad- vantage is mainly due to the fastText’s character level n-gram approach, which represents a mean- ingful performance difference both in training and …

Content truncated to 500 words