UoW: Multi-task Learning Gaussian Process for Semantic Textual Similarity

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 779–784, Dublin, Ireland, August 23-24, 2014. UoW: Multi-task Learning Gaussian Process for Semantic Textual Similarity Miguel Rios Research Group in Computational Linguistics University of Wolverhampton Stafford Street, Wolverhampton, WV1 1SB, UK M.Rios@wlv.ac.uk Lucia Specia Department of Computer Science University of Sheffield Regent Court, 211 Portobello, Sheffield, S1 4DP, UK L.Specia@sheffield.ac.uk Abstract We report results obtained by the UoW method in SemEval-2014’s Task 10 – Mul- tilingual Semantic Textual Similarity. We propose to model Semantic Textual Simi- larity in the context of Multi-task Learning in order to deal with inherent challenges of the task such as unbalanced performance across domains and the lack of training data for some domains (i.e. unknown domains). We show that the Multi-task Learning approach outperforms previous work on the 2012 dataset, achieves a ro- bust performance on the 2013 dataset and competitive results on the 2014 dataset. We highlight the importance of the chal- lenge of unknown domains, as it affects overall performance substantially. 1 Introduction The task of Semantic Textual Similarity (STS) (Agirre et al., 2012) is aimed at measuring the degree of semantic equivalence between a pair of texts. Natural Language Processing (NLP) ap- plications such as Question Answering (Lin and Pantel, 2001), Text Summarisation (Lin and Hovy, 2003) and Information Retrieval (Park et al., 2005) rely heavily on the ability to measure semantic similarity between pairs of texts. The STS eval- uation campaign provides datasets that consist of pairs of sentences from different NLP domains such as paraphrasing, video paraphrasing, and ma- chine translation (MT) evaluation. The participat- ing systems are required to predict a graded simi- larity score from 0 to 5, where a score of 0 means that the two sentences are on different topics and This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence details: http://creativecommons.org/licenses/by/4.0/ a score of 5 means that the two sentences have ex- actly the same meaning. Methods for STS are commonly based on com- puting various types of similarity metrics between the pair of sentences, where the similarity scores are used as features to train regression algorithms. B¨ar et al. (2012) use similarity metrics of vary- ing complexity. The range of features goes from simple string similarity metrics to complex vector space models. The method yielded the best av- erage results based on the official evaluation met- rics, despite not having achieved the best results in all individual domains. ˇSari´c et al. (2012) use a similar set up, extracting features from similarity metrics, where these features are based on word- overlap and syntax similarity. The method was among the best for domains related to paraphras- ing. It also achieved a high correlation between the training and test data. In contrast, for the ma- chine translation data the performance in the test set was lower than the one over the training data. A possible reason for the poor …

Content truncated to 500 words