SAARSHEFF at SemEval-2016 Task 1: Semantic Textual Similarity with Machine Translation Evaluation Metrics and (eXtreme) Boosted Tree Ensembles

Proceedings of SemEval-2016, pages 628–633, San Diego, California, June 16-17, 2016. c⃝2016 Association for Computational Linguistics SAARSHEFF at SemEval-2016 Task 1: Semantic Textual Similarity with Machine Translation Evaluation Metrics and (eXtreme) Boosted Tree Ensembles Liling Tan1, Carolina Scarton2, Lucia Specia2 and Josef van Genabith1, 3 Universit¨at des Saarlandes1 / Campus A2.2, Saarbr¨ucken, Germany University of Sheffield2 / Regent Court, 211 Portobello, Sheffield, UK Deutsches Forschungszentrum f¨ur K¨unstliche Intelligenz3 / Saarbr¨ucken, Germany alvations@gmail.com, c.scarton@sheffield.ac.uk, l.specia@sheffield.ac.uk, josef.van genabith@dfki.de Abstract This paper describes the SAARSHEFF sys- tems that participated in the English Seman- tic Textual Similarity (STS) task in SemEval- 2016. We extend the work on using ma- chine translation (MT) metrics in the STS task by automatically annotating the STS datasets with a variety of MT scores for each pair of text snippets in the STS datasets. We trained our systems using boosted tree ensembles and achieved competitive results that outperforms he median Pearson correlation scores from all participating systems. 1 Introduction Semantic Textual Similarity (STS) is the task of measuring the degree to which two texts have the same meaning (Agirre et al., 2014). For instance, given the two texts, “the man is slicing the tape from the box.” and “a man is cutting open a box.”, an STS system predicts a real number similarity score on a scale of 0 (no relation) to 5 (semantic equivalence). This paper presents a collaborative submission between Saarland University and University of Sheffield to the STS English shared task at SemEval- 2016. We have submitted three supervised models that predict the similarity scores for the STS task us- ing Machine Translation (MT) evaluation metrics as regression features. 2 Related Work Previous approaches have applied MT evaluation metrics for the STS task with progressively improv- ing results (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014; Agirre et al., 2015). At the pilot English STS-2012 task, Rios et al. (2012) trained a Support Vector Regressor using the lexical overlaps between the surface strings, named entities and semantic role labels and the BLEU (Pa- pineni et al., 2002) and METEOR (Banerjee and Lavie, 2005; Denkowski and Lavie, 2010) scores be- tween the text snippets and their best system scored a Pearson correlation mean of 0.3825. The system underperformed compared to the organizers’ base- line system1 which scored 0.4356. For the English STS-2013 task, Barr´on-Cede˜no et al. (2013) also used a Support Vector Regres- sor with an larger array of machine translation met- rics (BLEU, METEOR, ROUGE (Lin and Och, 2004), NIST (Doddington, 2002), TER (Snover et al., 2006)) with measures that compute sim- ilarities of dependency and constituency parses (Liu and Gildea, 2005) and semantic roles, dis- course representation and explicit semantic anal- ysis (Gabrilovich and Markovitch, 2007) annota- tions of the text snippets. These similarity mea- sures are packaged in the Asiya toolkit (Gim´enez and M`arquez, 2010). They scored 0.4037 mean score and performed better than the Takelab base- line (ˇSari´c et al., 2012) at 0.3639. At the SemEval-2014 Cross-level Semantic Simi- …

Content truncated to 500 words