OPI-JSA at SemEval-2017 Task 1: Application of Ensemble learning for computing semantic textual similarity

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 139–143, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics OPI-JSA at SemEval-2017 Task 1: Application of Ensemble learning for computing semantic textual similarity Martyna ´Spiewak, Piotr Sobecki and Daniel Kara´s National Information Processing Institute al. Niepodlego´sci 188b, 00-608 Warsaw, Poland {mspiewak, psobecki, dkaras}@opi.org.pl Abstract Semantic Textual Similarity (STS) evalua- tion assesses the degree to which two parts of texts are similar, based on their semantic evaluation. In this paper, we describe three models submitted to STS SemEval 2017. Given two English parts of a text, each of proposed methods outputs the assessment of their semantic similarity. We propose an approach for computing monolingual semantic textual similarity based on an ensemble of three distinct methods. Our model consists of recursive neural network (RNN) text auto-encoders ensemble with supervised a model of vec- torized sentences using reduced part of speech (PoS) weighted word embeddings as well as unsupervised a method based on word coverage (TakeLab). Addition- ally, we enrich our model with additional features that allow disambiguation of en- semble methods based on their efficiency. We have used Multi-Layer Perceptron as an ensemble classifier basing on estimations of trained Gradient Boosting Regressors. Results of our research proves that us- ing such ensemble leads to a higher ac- curacy due to a fact that each member- algorithm tends to specialize in particular type of sentences. Simple model based on PoS weighted Word2Vec word embed- dings seem to improve performance of more complex RNN based auto-encoders in the ensemble. In the monolingual English- English STS subtask our Ensemble based model achieved mean Pearson correlation of .785 compared with human annotators. 1 Introduction The objective of a system for evaluating seman- tic textual similarity, is to produce a value which serves as a rating of semantic similarity between pair of text samples. Such task certainly could not be regarded as toy problem, the results could be used to solve multiple real-world problems, e.g. plagiarism detection. We used described meth- ods in STS task in the SemEval 2017 competition (Bethard et al., 2017). 2 Methods 2.1 Data For the purpose of this research we have used datasets provided by the SemEval challenge or- ganizers containing English sentence pairs coming from several sources. STS Task objective is to produce a value in the range between 0.0 and 5.0, which assessing semantic similarity of a given pair of sentences. Intermediate levels are correspond- ing to partial similarity such as rough or topical equivalence but with differing details. In this study, we have used all English datasets provided by the challenge organizers until this year to train our su- pervised models. 2.2 Models The core of the system is based on widely used Gradient Boosting algorithm. The main novelty of described system lies in the formulation of its feature vectors. Each feature vector can be divided into two main parts: similarity scores and sentences’ descriptors. The process of feature extraction compiles simi- larity scores …

Content truncated to 500 words