Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations

Proceedings of COLING 2012: Technical Papers, pages 2109–2124, COLING 2012, Mumbai, December 2012. Improved Temporal Relation Classification using Dependency Parses and Selective Crowdsourced Annotations Jun − Ping NG Min − Yen KAN National University of Singapore, 13 Computing Drive, Singapore 117417 {junping,kanmy}@comp.nus.edu.sg ABSTRACT We study the problem of classifying the temporal relationship between events and time expres- sions in text. In contrast to previous methods that require extensive feature engineering, our approach is simple, relying only on a measure of parse tree similarity. Our method generates such tree similarity values using dependency parses as input to a convolution kernel. The resulting system outperforms the current state-of-the-art. To further improve classifier performance, we can obtain more annotated data. Rather than rely on expert annotation, we assess the feasibility of acquiring annotations through crowdsourcing. We show that quality temporal relationship annotation can be crowdsourced from novices. By leveraging the problem structure of temporal relation classification, we can selectively acquire annotations on problem instances that we assess as more difficult. Employing this annotation strategy allows us to achieve a classification accuracy of 73.2%, a statistically significant improvement of 8.6% over the previous state-of-the-art, while trimming annotation efforts by up to 37%. Finally, as we believe that access to sufficient training data is a significant barrier to current temporal relationship classification, we plan to share our collected data with the research community to promote benchmarking and comparative studies. KEYWORDS: temporal relations, information extraction, crowdsourcing, convolution kernels, clause structure, dependency parsing. 2109 1 Introduction We study the problem of temporal information extraction; specifically, we seek to establish when an event has taken place relative to a time expression. We refer to this task as event-temporal relationship classification. We adopt the same definitions for event and time expressions used in TimeML1. Consider the sentence2: Two top aides to Netanyahu , political adviser Uzi Arad and Cabinet Secretary Danny Naveh, left for Europe on Sunday , apparently to investigate the Syrian issue , the newspaper said. (1) In Sentence 1 “Sunday” is a time expression, while the various underlined words such as “said” are events. A pictorial summary of the timeline of the events as they are described in the sentence is shown in Figure 1. The time expression “Sunday” is marked on the timeline to represent an instance in time. The spans of the arrows representing each of the events denote the temporal relation between the three events and the time expression. In this case “left” happens in the same time span denoted by “Sunday”. The other two events (i.e. “investigate” and “said”) do not coincide with the “Sunday” marker and take place after the time span denoted by “Sunday”. Figure 1: Temporal relationships between events and a time expression in Sentence 1. Knowing such temporal relations are useful for downstream natural language processing applications. For example, Harabagiu and Bejan (2005) attempt to identify relationships between entities and time expressions to improve their question answering system. They combine knowledge of the relations between answer candidates and …

Content truncated to 500 words