Effective Use of Transformer Networks for Entity Tracking

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 759–769, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 759 Effective Use of Transformer Networks for Entity Tracking Aditya Gupta and Greg Durrett Department of Computer Science The University of Texas at Austin {agupta,gdurrett}@cs.utexas.edu Abstract Tracking entities in procedural language re- quires understanding the transformations aris- ing from actions on entities as well as those en- tities’ interactions. While self-attention-based pre-trained language encoders like GPT and BERT have been successfully applied across a range of natural language understanding tasks, their ability to handle the nuances of proce- dural texts is still untested. In this paper, we explore the use of pre-trained transformer net- works for entity tracking tasks in procedural text. First, we test standard lightweight ap- proaches for prediction with pre-trained trans- formers, and find that these approaches under- perform even simple baselines. We show that much stronger results can be attained by re- structuring the input to guide the transformer model to focus on a particular entity. Sec- ond, we assess the degree to which transformer networks capture the process dynamics, inves- tigating such factors as merged entities and oblique entity references. On two different tasks, ingredient detection in recipes and QA over scientific processes, we achieve state-of- the-art results, but our models still largely at- tend to shallow context clues and do not form complex representations of intermediate entity or process state.1 1 Introduction Transformer based pre-trained language models (Devlin et al., 2019; Radford et al., 2018, 2019; Joshi et al., 2019; Yang et al., 2019) have been shown to perform remarkably well on a range of tasks, including entity-related tasks like corefer- ence resolution (Kantor and Globerson, 2019) and named entity recognition (Devlin et al., 2019). This performance has been generally attributed to 1Code to reproduce experiments in this paper is available at https://github.com/aditya2211/ transformer-entity-tracking the robust transfer of lexical semantics to down- stream tasks. However, these models are still bet- ter at capturing syntax than they are at more entity- focused aspects like coreference (Tenney et al., 2019a,b); moreover, existing state-of-the-art ar- chitectures for such tasks often perform well look- ing at only local entity mentions (Wiseman et al., 2016; Lee et al., 2017; Peters et al., 2017) rather than forming truly global entity representations (Rahman and Ng, 2009; Lee et al., 2018). Thus, performance on these tasks does not form suffi- cient evidence that these representations strongly capture entity semantics. Better understanding the models’ capabilities requires testing them in do- mains involving complex entity interactions over longer texts. One such domain is that of pro- cedural language, which is strongly focused on tracking the entities involved and their interactions (Mori et al., 2014; Dalvi et al., 2018; Bosselut et al., 2018). This paper investigates the question of how transformer-based models form entity representa- tions and what these representations capture. We expect that after fine-tuning on a target task, …

Content truncated to 500 words