STS-UHH at SemEval-2017 Task 1: Scoring Semantic Textual Similarity Using Supervised and Unsupervised Ensemble

Proceedings of the 11th International Workshop on Semantic Evaluations (SemEval-2017), pages 175–179, Vancouver, Canada, August 3 - 4, 2017. c⃝2017 Association for Computational Linguistics STS-UHH at SemEval-2017 Task 1: Scoring Semantic Textual Similarity Using Supervised and Unsupervised Ensemble Sarah Kohail∗ LT Group CS Department Universit¨at Hamburg Amr Rekaby Salama∗ NATS Group CS Department Universit¨at Hamburg {kohail,salama,biemann}@informatik.uni-hamburg.de Chris Biemann LT Group CS Department Universit¨at Hamburg Abstract This paper reports the STS-UHH par- ticipation in the SemEval 2017 shared Task 1 of Semantic Textual Similarity (STS). Overall, we submitted 3 runs cov- ering monolingual and cross-lingual STS tracks. Our participation involves two ap- proaches: unsupervised approach, which estimates a word alignment-based sim- ilarity score, and supervised approach, which combines dependency graph sim- ilarity and coverage features with lexi- cal similarity measures using regression methods. We also present a way on ensem- bling both models. Out of 84 submitted runs, our team best multi-lingual run has been ranked 12th in overall performance with correlation of 0.61, 7th among 31 par- ticipating teams. 1 Introduction Semantic Textual Similarity (STS) measures the degree of semantic equivalence between a pair of sentences. Accurate estimation of semantic simi- larity would benefit many Natural Language Pro- cessing (NLP) applications such as textual entail- ment, information retrieval, paraphrase identifica- tion and plagiarism detection (Agirre et al., 2016). In an attempt to support the research efforts in STS, the SemEval STS shared Task (Agirre et al., 2017) offers an opportunity for developing cre- ative new sentence-level semantic similarity ap- proaches and to evaluate them on benchmark datasets. Given a pair of sentences, the task is to provide a similarity score on a scale of 0..5 ac- cording to the extent to which the two sentences are considered semantically similar, with 0 in- dicating that the semantics of the sentences are ∗*These authors contributed equally to this work completely unrelated and 5 signifying semantic equivalence. Final performance is measured by computing the Pearson’s correlation (ρ) between machine-assigned semantic similarity scores and gold standard scores provided by human annota- tors. Since last year, the STS task have been ex- tended to involve additional subtasks for cross- lingual STS. Similar to the monolingual STS task, the cross-lingual task requires the semantic sim- ilarity measurement for two snippets of text that are written in different languages. In contrast to last year’s edition (Agirre et al., 2016), the task is organized into 6 sub-tracks and a primary track, which is the average of all of the secondary sub-tracks results. Secondary sub-tracks involve scoring similarity for monolingual sentence pairs in one language (Arabic, English, Spanish), and cross-lingual sentence pairs from the combina- tion of two different languages (Arabic-English, Spanish-English, Turkish-English). Our paper proposes both supervised and unsuper- vised systems to automatically scoring semantic similarity between monolingual and cross-lingual short sentences. The two systems are then com- bined with an average ensemble to strengthen the similarity scoring performance. Out of 84 submis- sions, our system is placed 12th with an overall primary score of 0.61. 2 Related …

Content truncated to 500 words