SemEval-2013 Task 5: Evaluating Phrasal Semantics

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 39–47, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics SemEval-2013 Task 5: Evaluating Phrasal Semantics Ioannis Korkontzelos National Centre for Text Mining School of Computer Science University of Manchester, UK ioannis.korkontzelos@man.ac.uk Torsten Zesch UKP Lab, CompSci Dept. Technische Universit¨at Darmstadt Germany zesch@ukp.informatik.tu-darmstadt.de Fabio Massimo Zanzotto Department of Enterprise Engineering University of Rome “Tor Vergata” Italy zanzotto@info.uniroma2.it Chris Biemann FG Language Technology, CompSci Dept. Technische Universit¨at Darmstadt Germany biem@cs.tu-darmstadt.de Abstract This paper describes the SemEval-2013 Task 5: “Evaluating Phrasal Semantics”. Its first subtask is about computing the semantic simi- larity of words and compositional phrases of minimal length. The second one addresses deciding the compositionality of phrases in a given context. The paper discusses the impor- tance and background of these subtasks and their structure. In succession, it introduces the systems that participated and discusses evalu- ation results. 1 Introduction Numerous past tasks have focused on leveraging the meaning of word types or words in context. Exam- ples of the former are noun categorization and the TOEFL test, examples of the latter are word sense disambiguation, metonymy resolution, and lexical substitution. As these tasks have enjoyed a lot suc- cess, a natural progression is the pursuit of models that can perform similar tasks taking into account multiword expressions and complex compositional structure. In this paper, we present two subtasks de- signed to evaluate such phrasal models: a. Semantic similarity of words and compositional phrases b. Evaluating the compositionality of phrases in context For example, the first subtask addresses computing how similar the word “valuation” is to the compo- sitional sequence “price assessment”, while the sec- ond subtask addresses deciding whether the phrase ”piece of cake” is used literally or figuratively in the sentence “Labour was a piece of cake!”. The aim of these subtasks is two-fold. Firstly, considering that there is a spread interest lately in phrasal semantics in its various guises, they provide an opportunity to draw together approaches to nu- merous related problems under a common evalua- tion set. It is intended that after the competition, the evaluation setting and the datasets will comprise an on-going benchmark for the evaluation of these phrasal models. Secondly, the subtasks attempt to bridge the gap between established lexical semantics and full- blown linguistic inference. Thus, we anticipate that they will stimulate an increased interest around the general issue of phrasal semantics. We use the no- tion of phrasal semantics here as opposed to lexi- cal compounds or compositional semantics. Bridg- ing the gap between lexical semantics and linguis- tic inference could provoke novel approaches to cer- tain established tasks, such as lexical entailment and paraphrase identification. In addition, it could ul- 39 timately lead to improvements in a wide range of applications in natural language processing, such as document retrieval, clustering and classification, question answering, query expansion, synonym ex- traction, relation extraction, automatic translation, or textual advertisement matching in search …

Content truncated to 500 words