An Improved Neural Baseline for Temporal Relation Extraction

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6203–6209, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6203 An Improved Neural Baseline for Temporal Relation Extraction Qiang Ning,1 Sanjay Subramanian,2 and Dan Roth1,2 1University of Illinois at Urbana-Champaign 2University of Pennsylvania qning2@illinois.edu, {subs,danroth}@seas.upenn.edu Abstract Determining temporal relations (e.g., before or after) between events has been a challeng- ing natural language understanding task, partly due to the difficulty to generate large amounts of high-quality training data. Consequently, neural approaches have not been widely used on it, or showed only moderate improvements. This paper proposes a new neural system that achieves about 10% absolute improvement in accuracy over the previous best system (25% error reduction) on two benchmark datasets. The proposed system is trained on the state- of-the-art MATRES dataset and applies con- textualized word embeddings, a Siamese en- coder of a temporal common sense knowledge base, and global inference via integer linear programming (ILP). We suggest that the new approach could serve as a strong baseline for future research in this area. 1 Introduction Temporal relation (TempRel) extraction has been considered as a major component of understand- ing time in natural language (Do et al., 2012; Uz- Zaman et al., 2013; Minard et al., 2015; Llorens et al., 2015; Ning et al., 2018a). However, the an- notation process for TempRels is known to be time consuming and difficult even for humans, and ex- isting datasets are usually small and/or have low inter-annotator agreements (IAA); e.g., UzZaman et al. (2013); Chambers et al. (2014); O’Gorman et al. (2016) reported Kohen’s  and F1 in the 60’s. Albeit the significant progress in deep learning nowadays, neural approaches have not been used extensively for this task, or showed only moder- ate improvements (Dligach et al., 2017; Lin et al., 2017; Meng and Rumshisky, 2018). We think it is important for to understand: is it because we missed a “magic” neural architecture, because the training dataset is small, or because the quality of the dataset should be improved? Recently, Ning et al. (2018c) introduced a new dataset called Multi-Axis Temporal RElations for Start-points (MATRES). MATRES is still rela- tively small in its size (15K TempRels), but has a higher annotation quality from its improved task definition and annotation guideline. This paper uses MATRES to show that a long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) system can readily outperform the previ- ous state-of-the-art system, CogCompTime (Ning et al., 2018d), by a large margin. The fact that a standard LSTM system can significantly improve over a feature-based system on MATRES indicates that neural approaches have been mainly dwarfed by the quality of annotation, instead of specific neural architectures or the small size of data. To gain a better understanding of the standard LSTM method, we extensively compare the us- age of various word embedding techniques, in- cluding word2vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), FastText …

Content truncated to 500 words