UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 115–119, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics UdL at SemEval-2017 Task 1: Semantic Textual Similarity Estimation of English Sentence Pairs Using Regression Model over Pairwise Features Hussein T. Al-Natsheh 123, Lucie Martinet 124, Fabrice Muhlenbach 15, Djamel A. Zighed 12 1Universit´e de Lyon, 2Lyon 2, ERIC EA 3083, 5 Avenue Pierre Mend`es France, F69676 Bron Cedex - France 3CNRS, ISH FRE 3768, 14 Avenue Berthelot, F-69007 Lyon, France 4CESI EXIA/LINEACT, 19 Avenue Guy de Collongue, F-69130 ´Ecully, France 5UJM-Saint-Etienne, CNRS, Laboratoire Hubert Curien UMR 5516, F-42023 Saint Etienne, France hussein.al-natsheh@cnrs.fr, lucie.martinet@eric.univ-lyon2.fr fabrice.muhlenbach@univ-st-etienne.fr, djamel@zighed.com Abstract This paper describes the model UdL we proposed to solve the semantic textual similarity task of SemEval 2017 work- shop. The track we participated in was estimating the semantics relatedness of a given set of sentence pairs in English. The best run out of three submitted runs of our model achieved a Pearson correla- tion score of 0.8004 compared to a hid- den human annotation of 250 pairs. We used random forest ensemble learning to map an expandable set of extracted pair- wise features into a semantic similarity es- timated value bounded between 0 and 5. Most of these features were calculated us- ing word embedding vectors similarity to align Part of Speech (PoS) and Name En- tities (NE) tagged tokens of each sentence pair. Among other pairwise features, we experimented a classical tf–idf weighted Bag of Words (BoW) vector model but with character-based range of n-grams in- stead of words. This sentence vector BoW-based feature gave a relatively high importance value percentage in the fea- ture importances analysis of the ensemble learning. 1 Introduction Semantic Textual Similarity (STS) is a shared task that have been running every year by SemEval workshop since 2012. Each year, the participat- ing teams are encouraged to utilize the previous years data sets as a training set for their models. The teams are then ranked by their test score on a hidden human annotated pairs of sentences. Af- ter the end of the competition, the organizers pub- lish the gold standards and ask the teams of the coming year task to use it as a training set and so on. The description of STS2017 task is reported in (Cer et al., 2017). In STS2017 , the primary task consisted in 6 tracks covering both monolingual and cross-lingual sentence pairs for the languages Spanish, English, Arabic, and Turkish. Our team, UdL, only participated in the English monolingual track (Track 5). The data consist in thousands of pairs of sen- tences from various resources like (Twitter news, image captions, news headline, questions, an- swers, paraphrasing, post-editing...). For each pair, a human annotated score (from 0 to 5) is as- signed and indicates the semantic similarity values of the two sentences. The challenge is then to esti- mate the semantic similarity of 250 sentence pairs with hidden similarity values. The quality of the proposed models would …

Content truncated to 500 words