SOPA: Random Forests Regression for the Semantic Textual Similarity task

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 132–137, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SOPA: Random Forests Regression for the Semantic Textual Similarity task Davide Buscaldi, Jorge J. Garc´ıa Flores, Laboratoire d’Informatique de Paris Nord, CNRS (UMR 7030) Universit´e Paris 13, Sorbonne Paris Cit´e, Villetaneuse, France {buscaldi,jgflores}@lipn.univ-paris13.fr Ivan V. Meza and Isaac Rodr´ıguez Instituto de Investigaciones en Matemticas Aplicadas y en Sistemas (IIMAS) Universidad Nacional Autnoma de M´exico (UNAM) Ciudad Universitaria, DF, Mexico ivanvladimir,isaac@turing.iimas.unam.mx Abstract This paper describes the system used by the LIPN-IIMAS team in the Task 2, Semantic Textual Similarity, at SemEval 2015, in both the English and Spanish sub-tasks. We in- cluded some features based on alignment mea- sures and we tested different learning models, in particular Random Forests, which proved the best among those used in our participation. 1 Introduction Our participation in SemEval 2015 was focused on solving the technical problems that afflicted our previous participation (Buscaldi et al., 2014) and including additional features based on align- ments, such as the Sultan similarity (Sultan et al., 2014b) and the measure available in CMU Sphinx-4 (Lamere et al., 2003) for speech recognition. We baptised the new system SOPA from the Spanish word for “soup”, since it uses a heterogeneous mix of features. Well aware of the importance that the training corpus and the regression algorithms have for the STS task, we used language models to select the most appropriate training corpus for a given text, and we explored some alternatives to the ν-Support Vector Regression (ν-SVR) (Sch¨olkopf et al., 1999) used in our previous participations, specifically the Multi-Layer Perceptron (Bishop and others, 1995) and Random Forest (Breiman, 2001) regression al- gorithms. The obtained results show that Random Forests outperforms the other algorithms on every test set. We describe all the features in Section 2; the details on the learning algorithms and the training corpus selection process are described in Section 3, and the results obtained by the system are detailed in Section 4. 2 Similarity Measures In this section we describe the measures used as fea- tures in our system. The total number of features used was 16 in English and 14 in Spanish. Since most measures have already been used in our pre- vious participation, we provide only basic overview, referring the reader to the complete description in (Buscaldi et al., 2013) for further details. When POS tagging and NE recognition were required, we used the Stanford CoreNLP for English and Span- ish (Manning et al., 2014). 2.1 WordNet-based Conceptual Similarity This measure has been introduced in order to mea- sure similarities between concepts with respect to an ontology. The similarity is calculated as follows: first of all, words in sentences p and q are lemma- tised and mapped to the related WordNet synsets. All noun synsets are put into the set of synsets as- sociated to the sentence, Cp and Cq, respectively. If the synsets are in one of the other POS categories (verb, adjective, …

Content truncated to 500 words