LIPN-IIMAS at SemEval-2016 Task 1: Random Forest Regression Experiments on Align-and-Differentiate and Word Embeddings penalizing strategies

Proceedings of SemEval-2016, pages 726–731, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics LIPN-IIMAS at SemEval-2016 Task 1: Random Forest Regression Experiments on Align-and-Differentiate and Word Embeddings penalizing strategies Oscar Lithgow Centro de Ciencias Gen´omicas (CCG) Universidad Nacional Aut´onoma de M´exico (UNAM) Ciudad Universitaria, DF, Mexico owl@turing.iimas.unam.mx Ivan V. Meza, Albert Orozco Instituto de Investigaciones en Matem´aticas Aplicadas y en Sistemas Universidad Nacional Aut´onoma de M´exico (UNAM) Ciudad Universitaria, DF, Mexico {ivanvladimir,albert}@turing.iimas.unam.mx Jorge Gacia Flores, Davide Buscaldi Laboratoire d’Informatique de Paris Nord, CNRS (UMR 7030) Universit´e Paris 13, Sorbonne Paris Cit´e, Villetaneuse, France {buscaldi,jgflores}@lipn.univ-paris13.fr Abstract This paper describes the SOPA-N system used by the LIPN-IIMAS team in Semeval 2016 Semantic Textual Similarity (Task 1). We based our work on the SOPA 2015 system. The SOPA-2015 system used 16 similarity features (including Wordnet, Information Re- trieval and Syntactic Dependencies) within a Random Forest learning model. We expanded this system with an Align and Differentiate based strategy, word embeddings and penal- ization, which showed 6.8% of improvement on the development set. However, we found that on the evaluation data for the 2016 STS shared task, the 2015 system outperformed our newer systems. 1 Introduction The SOPA system combines a regression model with multi-level similarity measures, from very simple ones (like edit distance) to more sophisticated ones (like IR-based or Wordnet similarity) (Buscaldi et al., 2013). Our goal this year was to add an align- and-differentiate penalizing strategy based on (Han et al., 2015) to our 2015 system (SOPA) (Buscaldi et al., 2015). The previous version consists on 16 similarity features which are regressed using a Ran- dom Forest learning algorithm. The rationale of the penalization was to account for cases when appar- ent distributional alignments are closer than their semantically equivalent (for instance, colors: while black and white are distributional close because are colors, but they are not semantically equivalent). We present two versions of the enhanced system SOPA 100 and 1000 which refers to the number of estima- tion trees used by the Random Forest algorithm. We find that augmenting our 2015 model with an align and differentiate module boosts performance on the 2015 evaluation data. However, on the STS 2016 test data it only outperformed our previous ap- proach on the plagiarism dataset. A closer error analysis showed that the gap between SOPA 100 and the gold standard (GS) scores are systemati- cally positive, meaning that our new system over- estimated the semantic similarity between phrases. Another interesting finding from the error analysis was that in those sentences where SOPA 100 out- perform SOPA, it was also more accurate, given the fact that the SOPA 100 standard deviation from the gold standard annotation was smaller than the one of 726 SOPA. Ablation tests for every feature were performed as well. Within our 2015 system, Sultan’s align- ment based similarity feature, Sultan et al. (2015)’s feature 18 , seems to be pulling down scores both for the headlines and question-question domains of 2016 dataset, despite the fact that …

Content truncated to 500 words