USFD: Preliminary Exploration of Features and Classifiers for the TempEval-2007 Task

Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), pages 438–441, Prague, June 2007. c⃝2007 Association for Computational Linguistics USFD: Preliminary Exploration of Features and Classifiers for the TempEval-2007 Tasks Mark Hepple Dept of Computer Science University of Sheffield Regent Court 211 Portobello Street Sheffield S1 4DP, UK hepple@dcs.shef.ac.uk Andrea Setzer Dept of Computer Science University of Sheffield Regent Court 211 Portobello Street Sheffield S1 4DP, UK andrea@dcs.shef.ac.uk Rob Gaizauskas Dept of Computer Science University of Sheffield Regent Court 211 Portobello Street Sheffield S1 4DP, UK robertg@dcs.shef.ac.uk Abstract We describe the Sheffield system used in TempEval-2007. Our system takes a machine-learning (ML) based approach, treating temporal relation assignment as a simple classification task and using features easily derived from the TempEval data, i.e. which do not require ‘deeper’ NLP analy- sis. We aimed to explore three questions: (1) How well would a ‘lite’ approach of this kind perform? (2) Which features con- tribute positively to system performance? (3) Which ML algorithm is better suited for the TempEval tasks? We used the Weka ML workbench to facilitate experimenting with different ML algorithms. The paper de- scribes our system and supplies preliminary answers to the above questions. 1 Introduction The Sheffield team were involved in TempEval as co-proposers/co-organisers of the task.1 For our par- ticipation in the task, we decided to pursue an ML- based approach, the benefits of which have been ex- plored elsewhere (Boguraev and Ando, 2005; Mani et al., 2006). For the TempEval tasks, this is easily done by treating the assignment of temporal relation types as a simple classification task, using readily available information for the instance features. More specifically, the features used were ones provided as 1We maintained a strict separation between persons assisting in annotation of the test corpus and those involved in system development. attributes in the TempEval data annotation for the events/times being related, plus some additional fea- tures that could be straightforwardly computed from documents, i.e. without the use of more heavily ‘en- gineered’ NLP components. The aims of this work were three-fold. First, we wanted to see whether a ‘lite’ approach of this kind could yield reasonable performance, before pursuing possibilities that re- lied on using ‘deeper’ NLP analysis methods. Sec- ondly, we were interested to see which of the fea- tures considered would contribute positively to sys- tem performance. Thirdly, rather than selecting a single ML approach (e.g. one of those currently in vogue within NLP), we wanted to look across ML algorithms to see if any approach was better suited to the TempEval tasks than any other, and conse- quently we used the Weka workbench (Witten and Frank, 2005) in our ML experiments. In what follows, we will first describe how our system was constructed, before going on to discuss our main observations around the key aims men- tioned above. For example, in regard to our ‘lite’ ap- proach, we would observe (c.f. the results reported in the Task Description paper) that although some other systems scored more highly, …

Content truncated to 500 words