IIRG: A Naive Approach to Evaluating Phrasal Semantics

Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Seventh International Workshop on Semantic Evaluation (SemEval 2013), pages 103–107, Atlanta, Georgia, June 14-15, 2013. c⃝2013 Association for Computational Linguistics IIRG: A Na¨ıve Approach to Evaluating Phrasal Semantics Lorna Byrne, Caroline Fenlon, John Dunnion School of Computer Science and Informatics University College Dublin Ireland {lorna.byrne@ucd.ie,caroline.fenlon@ucdconnect.ie,john.dunnion@ucd.ie} Abstract This paper describes the IIRG 1 system en- tered in SemEval-2013, the 7th International Workshop on Semantic Evaluation. We partic- ipated in Task 5 Evaluating Phrasal Semantics. We have adopted a token-based approach to solve this task using 1) Na¨ıve Bayes methods and 2) Word Overlap methods, both of which rely on the extraction of syntactic features. We found that the word overlap method signifi- cantly out-performs the Na¨ıve Bayes methods, achieving our highest overall score with an ac- curacy of approximately 78%. 1 Introduction The Phrasal Semantics task consists of two related subtasks. Task 5A requires systems to evaluate the semantic similarity of words and compositional phrases. Task 5B requires systems to evaluate the compositionality of phrases in context. We partici- pated in Task 5B and submitted three runs for eval- uation, two runs using the Na¨ıve Bayes Machine Learning Algorithm and a Word Overlap run using a simple bag-of-words approach. Identifying non-literal expressions poses a major challenge in NLP because they occur frequently and often exhibit irregular behavior by not adhering to grammatical constraints. Previous research in the area of identifying literal/non-literal use of expres- sions includes generating a wide range of different features for use with a machine learning prediction algorithm. (Li and Sporleder, 2010) present a system 1Intelligent Information Retrieval Group involving identifying the global and local contexts of a phrase. Global context was determined by look- ing for occurrences of semantically related words in a given passage, while local context focuses on the words immediately preceding and following the phrase. Windows of five words at each side of the target were taken as features. More syntactic fea- tures were also used, including details of nodes from the dependency tree of each example. The system produced approximately 90% accuracy when tested, for both idiom-specific and generic models. It was found that the statistical features (global and local contexts) performed well, even on unseen phrases. (Katz and Giesbrecht, 2006) found that similarities between words in the expression and its context indi- cate literal usage. This is comparable to (Sporleder and Li, 2009), which used cohesion-based classi- fiers based on lexical chains and graphs. Unsu- pervised approaches to classifying idiomatic use in- clude clustering (Fazly et al., 2009), which classified data based on semantic analyzability (whether the meaning of the expression is similar to the mean- ings of its parts) and lexical and syntactic flexibility (measurements of how much variation exists within the expression). 2 Task 5B In Task 5B, participants were required to make a binary decision as to whether a target phrase is used figuratively or literally within a given context. The phrase “drop the ball” can be used …

Content truncated to 500 words