SEF@UHH at SemEval-2017 Task 1: Unsupervised Knowledge-Free Semantic Textual Similarity via Paragraph Vector

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 170–174, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics SEF@UHH at SemEval-2017 Task 1: Unsupervised Knowledge-Free Semantic Textual Similarity via Paragraph Vector Mirela-Stefania Duma and Wolfgang Menzel University of Hamburg Natural Language Systems Division {mduma, menzel}@informatik.uni-hamburg.de Abstract This paper describes our unsupervised knowledge-free approach to the SemEval- 2017 Task 1 Competition. The proposed method makes use of Paragraph Vector for assessing the semantic similarity between pairs of sentences. We experimented with various dimensions of the vector and three state-of-the-art similarity metrics. Given a cross-lingual task, we trained models cor- responding to its two languages and com- bined the models by averaging the simi- larity scores. The results of our submitted runs are above the median scores for five out of seven test sets by means of Pear- son Correlation. Moreover, one of our sys- tem runs performed best on the Spanish- English-WMT test set ranking first out of 53 runs submitted in total by all partici- pants. 1 Introduction Semantic Textual Similarity (STS) aims to assess the degree to which two snippets of text are re- lated in meaning to each other. The SemEval an- nual competition offers a track on STS (Cer et al., 2017) where submitted STS systems are evaluated in terms of the Pearson correlation between ma- chine assigned semantic similarity scores and hu- man judgments. We participated in both monolingual sub-tracks and cross-lingual sub-tracks. Given a sentence pair in the same language, the SemEval STS task is to assign a similarity score to it ranging from 0 to 5, with 0 implying that the semantics of the sen- tences are completely independent and 5 denoting semantic equivalence (Cer et al., 2017). The cross- lingual side of STS is similar to the initial task, but differs in the input sentences which come from two languages. This year’s shared task features six sub-tasks: Arabic-Arabic, Arabic-English, Spanish-Spanish, Spanish-English (two test sets), English-English and a surprise task (Turkish-English) for which no annotated data is offered. For example, for the English monolingual STS track, the pair of sentences below had a score of 3 assigned by human annotators, meaning that the two sentences are roughly equivalent, but some es- sential information differs or is missing (Cer et al., 2017). Bayes’ theorem was named after Rev Thomas Bayes and is a method used in probability theory. As an official theorem, Bayes’ theorem is valid in all universal interpretations of probability. We present an unsupervised, knowledge-free approach that utilizes Paragraph Vector (Le and Mikolov, 2014) to represent sentences by means of continuous distributed vectors. In addition to experimenting with feature spaces of different dimensionality, we also compare three state-of-the-art similarity metrics (Cosine, Bray-Curtis and Correlation) for calculating the STS scores. We do not make use of any lexical or semantic resources, nor hand-annotated labeled corpora in addition to the distributed representa- tions trained on non-annotated text. The approach gives promising results on all sub-tasks, with our submitted systems …

Content truncated to 500 words