SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals

Proceedings of the 5th International Workshop on Semantic Evaluation, ACL 2010, pages 33–38, Uppsala, Sweden, 15-16 July 2010. c⃝2010 Association for Computational Linguistics SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations Between Pairs of Nominals Iris Hendrickx∗, Su Nam Kim† , Zornitsa Kozareva‡ , Preslav Nakov§ , Diarmuid ´O S´eaghdha¶, Sebastian Pad´o∥, Marco Pennacchiotti∗∗, Lorenza Romano††, Stan Szpakowicz‡‡ Abstract SemEval-2 Task 8 focuses on Multi-way classification of semantic relations between pairs of nominals. The task was designed to compare different approaches to seman- tic relation classification and to provide a standard testbed for future research. This paper defines the task, describes the train- ing and test data and the process of their creation, lists the participating systems (10 teams, 28 runs), and discusses their results. 1 Introduction SemEval-2010 Task 8 focused on semantic rela- tions between pairs of nominals. For example, tea and ginseng are in an ENTITY-ORIGIN relation in “The cup contained tea from dried ginseng.”. The automatic recognition of semantic relations has many applications, such as information extraction, document summarization, machine translation, or construction of thesauri and semantic networks. It can also facilitate auxiliary tasks such as word sense disambiguation, language modeling, para- phrasing, and recognizing textual entailment. Our goal was to create a testbed for automatic classification of semantic relations. In developing the task we met several challenges: selecting a suitable set of relations, specifying the annotation procedure, and deciding on the details of the task itself. They are discussed briefly in Section 2; see also Hendrickx et al. (2009), which includes a sur- vey of related work. The direct predecessor of Task 8 was Classification of semantic relations between nominals, Task 4 at SemEval-1 (Girju et al., 2009), ∗University of Lisbon, iris@clul.ul.pt † University of Melbourne, snkim@csse.unimelb.edu.au ‡ Information Sciences Institute/University of Southern California, kozareva@isi.edu § National University of Singapore, nakov@comp.nus.edu.sg ¶University of Cambridge, do242@cl.cam.ac.uk ∥University of Stuttgart, pado@ims.uni-stuttgart.de ∗∗Yahoo! Inc., pennacc@yahoo-inc.com ††Fondazione Bruno Kessler, romano@fbk.eu ‡‡University of Ottawa and Polish Academy of Sciences, szpak@site.uottawa.ca which had a separate binary-labeled dataset for each of seven relations. We have defined SemEval- 2010 Task 8 as a multi-way classification task in which the label for each example must be chosen from the complete set of ten relations and the map- ping from nouns to argument slots is not provided in advance. We also provide more data: 10,717 an- notated examples, compared to 1,529 in SemEval-1 Task 4. 2 Dataset Creation 2.1 The Inventory of Semantic Relations We first decided on an inventory of semantic rela- tions. Ideally, it should be exhaustive (enable the description of relations between any pair of nomi- nals) and mutually exclusive (each pair of nominals in context should map onto only one relation). The literature, however, suggests that no relation inven- tory satisfies both needs, and, in practice, some trade-off between them must be accepted. As a pragmatic compromise, we selected nine relations with coverage sufficiently broad to be of general and practical interest. We aimed at avoid- ing semantic overlap as much as …

Content truncated to 500 words