SemEval-2015 Task 5: QA TempEval - Evaluating Temporal Information Understanding with Question Answering

Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), pages 792–800, Denver, Colorado, June 4-5, 2015. c⃝2015 Association for Computational Linguistics SemEval-2015 Task 5: QA TEMPEVAL - Evaluating Temporal Information Understanding with Question Answering Hector Llorens♣, Nathanael Chambers♦, Naushad UzZaman♣, Nasrin Mostafazadeh✶, James Allen✶, James Pustejovsky♠ ♣ Nuance Communications, USA ♦ United States Naval Academy, USA ✶ University of Rochester, USA ♠ Brandeis University, USA hector.llorens@nuance.com, nchamber@usna.edu Abstract QA TempEval shifts the goal of previous TempEvals away from an intrinsic evaluation methodology toward a more extrinsic goal of question answering. This evaluation requires systems to capture temporal information rele- vant to perform an end-user task, as opposed to corpus-based evaluation where all temporal information is equally important. Evaluation results show that the best automated TimeML annotations reach over 30% recall on ques- tions with ‘yes’ answer and about 50% on eas- ier questions with ‘no’ answers. Features that helped achieve better results are event corefer- ence and a time expression reasoner. 1 Introduction QA TempEval is a follow up of the TempEval series in SemEval: TempEval-1 (Verhagen et al., 2007), TempEval-2 (Verhagen et al., 2010), and TempEval- 3 (UzZaman et al., 2013). TempEval focuses on evaluating systems that extract temporal expressions (timexes), events, and temporal relations as defined in the TimeML standard (Pustejovsky et al., 2003) (timeml.org). QA TempEval is unique in its focus on evaluating temporal information that directly ad- dress a QA task. TimeML was originally developed to support research in complex temporal QA within the field of artificial intelligence (AI). However, de- spite its original goal, the complexity of temporal QA has caused most research on automatic TimeML systems to focus on a more straightforward temporal information extraction (IE) task. QA TempEval still requires systems to extract temporal relations just like previous TempEvals, however, the QA evalua- tion is solely based on how well the relations answer questions about the documents. It is no longer about annotation accuracy, but rather the accuracy for tar- geted questions. Not only does QA represent a more natural way to evaluate temporal information understanding (Uz- Zaman et al., 2012), but also annotating docu- ments with question sets requires much less exper- tise and effort for humans than corpus-based evalua- tion which requires full manual annotation of tempo- ral information. In QA TempEval a document does not require the markup of all the temporal entities and relations, but rather a markup of a few key re- lations central to the text. Although the evaluation schema changes in QA TempEval, the task for par- ticipating systems remains the same: extracting tem- poral information from plain text documents. Here we re-use TempEval-3 task ABC, where sys- tems are required to perform end-to-end TimeML annotation from plain text, including the complete set of temporal relations (Allen, 1983). However, unlike TempEval-3, there are no subtasks focusing on specific elements (such as an event extraction evaluation). Also, instead of IE performance mea- surement for evaluation, a QA performance (on a set of human-created temporal questions …

Content truncated to 500 words