Towards Multilingual Event Extraction Evaluation: A Case Study for the Czech Language

Proceedings of Recent Advances in Natural Language Processing, pages 627–635, Hissar, Bulgaria, Sep 7–9 2015. Towards Multilingual Event Extraction Evaluation: A Case Study for the Czech Language Josef Steinberger DCSE, NTIS Centre, FAV University of West Bohemia Univerzini 8, 306 14 Plzen Czech Republic jstein@kiv.zcu.cz Hristo Tanev Joint Research Centre European Commission via Fermi 2749, Ispra Italy hristo.tanev@jrc.ec.europa.eu Abstract This paper presents a multilingual corpus of news, annotated with event metadata in- formation. The events in our corpus are from the domain of violence, natural and man made disasters. The main goal of the corpus is automatic evaluation of event de- tection and extraction systems in different languages. As a use case, we take a rule- based event extraction system, extend it to cover a new language, Czech in our case, and evaluate it on the corpus. We explain what needs to be done to cover a new lan- guage, especially learning domain-specific dictionaries and event extraction patterns. The evaluation of the Czech system can be viewed as a starting point for further re- search into the evaluation of multilingual event extraction systems, which is an im- portant stage during the development of such systems. The comparison of the per- formance for the Czech and English sys- tems indicates the importance for multilin- gual event extraction evaluation. 1 Introduction The quantity of information on Internet has reached a critical point. Simple keyword indexing cannot satisfy any more the need for fast and accu- rate access to this information ocean. In this light, the development of effective methods for informa- tion extraction are of particular importance. In this paper we will discuss issues related to automatic event metadata extraction. Mainstream media and part of the social media are event-oriented, there- fore development of methods for accurate identi- fication, classification and extraction of metadata about events is of particular importance. Note- worthy, crisis events, such as natural, man-made disasters, crime and armed conflicts are the most frequent types of events, described in online news and often referred to in social media. Due to the complexity of the event extraction task, preparing a gold standard and evaluation of event extraction systems is not straightforward. Event annotation can be done in many different ways. Different taxonomies of event types can be used, as well as different event properties may be annotated. Moreover, one cannot give a sin- gle accuracy number, which characterizes an event extraction system performance. Rather than that, the accuracy for the extraction of each event prop- erty is measured separately. Even measuring the overlap between the gold standard and the output, produced by a system, can be done in different ways. Similarly, evaluating the similarity between event types, such as bombing and terrorist attack requires investigating into the nature of the events and the goals of the evaluation schema. In this paper we make a small step into the in- finite field of problems and solutions which the evaluation of event extraction system poses in front of the researchers in the …

Content truncated to 500 words