Temporal Information Annotation: Crowd vs. Experts

Temporal Information Annotation: Crowd vs. Experts Tommaso Caselli∗, Rachele Sprugnoli, Oana Inel∗♣ ∗The Network Institute Vrije Universiteit Amsterdam; FBK and University of Trento; ♣IBM Nederland, CAS Benelux De Boelelaan 1105 1081 HV Amsterdam; Via Sommarive, 18 38122 Povo (TN); Johan Huizingalaan 765, 1066 VH Amsterdam {t.caselli;oana.inel}@vu.nl∗, sprugnoli@fbk.eu Abstract This paper describes two sets of crowdsourcing experiments on temporal information annotation conducted on two languages, i.e., English and Italian. The first experiment, launched on the CrowdFlower platform, was aimed at classifying temporal relations given target entities. The second one, relying on the CrowdTruth metric, consisted in two subtasks: one devoted to the recognition of events and temporal expressions and one to the detection and classification of temporal relations. The outcomes of the experiments suggest a valuable use of crowdsourcing annotations also for a complex task like Temporal Processing. Keywords: crowdsourcing, temporal processing, corpus creation 1. Introduction This paper reports on a set of crowdsourcing experiments on temporal information annotation whose goal is to com- pare the results of experts with those of the crowd for the annotation of temporal expressions, events and temporal re- lations. The results will allow us to gain better insights on the Temporal Processing task by suggesting changes in the current annotation practices. The TimeML Annotation Guidelines (Pustejovsky et al., 2003a) and the TempEval evaluation campaigns1 have greatly contributed to the development of TimeML- compliant annotation schemes in different languages and the setting of evaluation procedures for systems against public benchmark data. Reviewing the performance of systems shows that Tempo- ral Processing is not a trivial task, especially when dealing with temporal relations. In Table 1 we report the results of the best systems for Italian and English with respect to four tasks: a.) Temporal Expressions (TIMEXes) Identifi- cation; b.) Event Extraction; c.) Temporal Relation Iden- tification and Classification from raw text (TLINKs raw); and d.) Temporal Relation Classification given Gold en- tities (TLINKs Gold). The figures have been extracted from TempEval-3 for English (UzZaman et al., 2013) and EVENTI for Italian (Caselli et al., 2014). Although a direct comparison among the test sets cannot be done, the difference in the systems’ performance in the two languages can hardly be explained by making refer- ence only to language specific issues. Furthermore, the systems’ performance seems to be also affected by the lan- guage specific annotation guidelines and the quality of the annotated corpora. Concerning the annotation guidelines, the two languages: a.) share the same annotation philoso- phy (i.e., adherence to the superficial text form), the same set of markables and values; b.) are compliant with the ISO-TimeML standard; c.) have benefited from a continu- 1TempEval 2007 (Verhagen et al., 2007): http://www. timeml.org/tempeval/; TempEval 2010 (Verhagen et al., 2010): http://www.timeml.org/tempeval2/; TempEval 2013: http://www.cs.york.ac.uk/ semeval-2013/task1/ Task Lang IAA System F1 TIMEXes IT P&R=0.95 HeidelTime1.8 0.709 EN P&R=0.83 HeidelTime-t 0.776 EVENTs IT P&R=0.86 FBK-B1 0.867 EN P&R=0.78 ATT-1 0.81 TLINK Raw IT Dice=0.86 FBK C1 0.264 EN P&R=0.55 ClearTK-2 0.309 TLINK Gold IT K=0.88 FBK D1 0.736 EN K=0.71 UTTime-1,4 …

Content truncated to 500 words