USAAR-SHEFFIELD: Semantic Textual Similarity with Deep Regression and Machine Translation Evaluation Metrics

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 85–89, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics USAAR-SHEFFIELD: Semantic Textual Similarity with Deep Regression and Machine Translation Evaluation Metrics Liling Tanα, Carolina Scartonβ, Lucia Speciaβ and Josef van Genabithγ αUniversit¨at des Saarlandes / Campus A2.2, Saarbr¨ucken, Germany βUniversity of Sheffield / Regent Court, 211 Portobello, Sheffield, UK γDeutsches Forschungszentrum f¨ur K¨unstliche Intelligenz / Saarbr¨ucken, Germany alvations@gmail.com, c.scarton@sheffield.ac.uk, l.specia@sheffield.ac.uk, josef.van genabith@dfki.de Abstract This paper describes the USAAR- SHEFFIELD systems that participated in the Semantic Textual Similarity (STS) English task of SemEval-2015. We extend the work on using machine translation evaluation metrics in the STS task. Different from previous approaches, we regard the metrics’ robustness across different text types and conflate the training data across different subcorpora. In addition, we introduce a novel deep regressor architecture and evaluated its efficiency in the STS task. 1 Introduction Semantic Textual Similarity (STS) is the task of measuring the degree to which two text snippets have the same meaning (Agirre et al., 2014). For instance, given the two texts, ”a dog sprints across the water” and ”a dog jumps through water”, partic- ipating systems are required to predict a real number similarity score on a scale of 0 (no relation) to 5 (se- mantic equivalence). This paper presents a collaborative submission between Saarland University and University of Sheffield to the STS English shared task at SemEval- 2015. We have submitted three models that use Ma- chine Translation (MT) evaluation metrics as fea- tures to build supervised regressors that predict the similarity scores for the STS task. We introduce two variants of a novel deep regressor architecture and a classical baseline regression system that uses MT evaluation metrics as input features. 2 Related Work Previously, research teams have applied MT evalua- tion metrics for the STS task with increasingly bet- ter results (Agirre et al., 2012; Agirre et al., 2013; Agirre et al., 2014). Rios et al. (2012) trained a Support Vector Regressor scoring a Pearson corre- lation mean of 0.3825 (Baseline1: 0.4356). Barr´on- Cede˜no et al. (2013) also used a Support Vector Re- gressor and did better than the baseline at 0.4037 mean score (Baseline: 0.3639). Huang and Chang (2014) used a linear regressor and scored 0.792 beat- ing the baseline system (Baseline: 0.613). Another notable mention of MT technology in the STS task is the use of referential translation ma- chines to predict and derive features instead of us- ing MT evaluation metrics (Bic¸ici and van Genabith, 2013; Bic¸ici and Way, 2014). These previous approaches have trained a differ- ent system for each subcorpus provided by the task organizers. We have chosen to combine the differ- ent subcorpora since MT evaluation metrics are ex- pected to be robust against text types and domains (Han et al., 2012; Pad´o et al., 2009). Much of the previous work on using MT evalu- ation metrics is based on improving the regressors through algorithm choice, feature selection and pa- rameters tuning. We …

Content truncated to 500 words