Meerkat Mafia: Multilingual and Cross-Level Semantic Textual Similarity Systems

Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 416–423, Dublin, Ireland, August 23-24, 2014. Meerkat Mafia: Multilingual and Cross-Level Semantic Textual Similarity Systems Abhay Kashyap, Lushan Han, Roberto Yus, Jennifer Sleeman, Taneeya Satyapanich, Sunil Gandhi and Tim Finin University of Maryland, Baltimore County Baltimore, MD 21250 USA {abhay1,lushan1,ryus,jsleem1,taneeya1,sunilga1,finin}@umbc.edu Abstract We describe UMBC’s systems developed for the SemEval 2014 tasks on Multi- lingual Semantic Textual Similarity (Task 10) and Cross-Level Semantic Similarity (Task 3). Our best submission in the Multilingual task ranked second in both English and Spanish subtasks using an unsupervised approach. Our best sys- tems for Cross-Level task ranked second in Paragraph-Sentence and first in both Sentence-Phrase and Word-Sense subtask. The system ranked first for the Phrase- Word subtask but was not included in the official results due to a late submission. 1 Introduction We describe the semantic text similarity systems we developed for two of the SemEval tasks for the 2014 International Workshop on Semantic Evalu- ation. We developed systems for task 3, Cross- Level Semantic Similarity (Jurgens et al., 2014), and task 10, Multilingual Semantic Textual Simi- larity (Agirre et al., 2014). A key component in all the systems was an enhanced version of the word similarity system used in our entry (Han et al., 2013b) in the 2013 SemEval Semantic Textual Similarity task. Our best system in the Multilingual Semantic Textual Similarity task used an unsupervised ap- proach and ranked second in both the English and Spanish subtasks. In the Cross-Level Semantic Similarity task we developed a number of new al- gorithms and used new linguistic data resources. In this task, our best systems ranked second in the Paragraph-Sentence task, first in the Sentence- Phrase task and first in the Word-Sense task. The This work is licensed under a Creative Commons At- tribution 4.0 International Licence. Page numbers and pro- ceedings footer are added by the organisers. Licence de- tails:http://creativecommons.org/licenses/by/4.0/ system ranked first for the Phrase-Word task but was not included in the official results due to a late submission. The remainder of the paper proceeds as follows. Section 2 describes our word similarity model and it’s wrapper to deal with named entities and out of vocabulary words. Sections 3 and 4 describe how we extended the word similarity model for the specific tasks. Section 5 presents the results we achieved on these tasks along with instances where the system failed. Section 6 highlights our future plans for improving the system. 2 Semantic Word Similarity Model 2.1 LSA Word Similarity Model Our word similarity model is a revised version of the one we used in the 2013 *SEM semantic text similarity task. This was in turn derived from a system developed for the Graph of Relations project (UMBC, 2013b). For SemEval, we wanted a measure that considered a word’s semantics but not its lexical category, e.g., the verb “marry” should be semantically similar to the noun “wife”. An online demonstration of a similar model de- veloped for the GOR project is available …

Content truncated to 500 words