SemEval-2015 Task 15: A CPA dictionary-entry-building task

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 315–324, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 15: A Corpus Pattern Analysis Dictionary-Entry-Building Task V´ıt Baisa Masaryk University xbaisa@fi.muni.cz Jane Bradbury University of Wolverhampton J.Bradbury3@wlv.ac.uk Silvie Cinkov´a Charles University cinkova@ufal.mff.cuni.cz Isma¨ıl El Maarouf University of Wolverhampton i.el-maarouf@wlv.ac.uk Adam Kilgarriff Lexical Computing Ltd adam@sketchengine.co.uk Octavian Popescu IBM Research Center o.popescu@us.ibm.com Abstract This paper describes the first SemEval task to explore the use of Natural Language Process- ing systems for building dictionary entries, in the framework of Corpus Pattern Analysis. CPA is a corpus-driven technique which pro- vides tools and resources to identify and rep- resent unambiguously the main semantic pat- terns in which words are used. Task 15 draws on the Pattern Dictionary of English Verbs (www.pdev.org.uk), for the targeted lexi- cal entries, and on the British National Corpus for the input text. Dictionary entry building is split into three subtasks which all start from the same con- cordance sample: 1) CPA parsing, where ar- guments and their syntactic and semantic cate- gories have to be identified, 2) CPA clustering, in which sentences with similar patterns have to be clustered and 3) CPA automatic lexicog- raphy where the structure of patterns have to be constructed automatically. Subtask 1 attracted 3 teams, though none could beat the baseline (rule-based system). Subtask 2 attracted 2 teams, one of which beat the baseline (majority-class classifier). Sub- task 3 did not attract any participant. The task has produced a major semantic multi- dataset resource which includes data for 121 verbs and about 17,000 annotated sentences, and which is freely accessible. 1 Introduction It is a central vision of NLP to represent the mean- ings of texts in a formalised way, amenable to au- tomated reasoning. Since its birth, SEMEVAL (or SENSEVAL as it was then; (Kilgarriff and Palmer, 2000)) has been part of the programme of enrich- ing NLP analyses of text so they get ever closer to a ’meaning representation’. In relation to lexi- cal information, this meant finding a lexical resource which • identified the different meanings of words in a way that made high-quality disambiguation possible, • represented those meanings in ways that were useful for the next steps of building meaning representations. Most lexical resources explored to date have had only limited success, on either front. The most obvious candidates—published dictionaries and WordNets—look like they might support the first task, but are very limited in what they offer to the second. FrameNet moved the game forward a stage. Here was a framework with a convincing account of how the lexical entry might contribute to building the meaning of the sentence, and with enough meat in the lexical entries (e.g. the verb frames) so that it might support disambiguation. Papers such as (Gildea and Jurafsky, 2002) looked promising, and in 2007 there was a SEMEVAL task on Frame Se- mantic Structure Extraction (Baker et al., 2007) and in 2010, one on Linking Events and …

Content truncated to 500 words