Time-Stamped Language Model: Teaching Language Models to Understand The Flow of Events

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4560–4570 June 6–11, 2021. ©2021 Association for Computational Linguistics 4560 Time-Stamped Language Model: Teaching Language Models to Understand the Flow of Events Hossein Rajaby Faghihi Michigan State University rajabyfa@msu.edu Parisa Kordjamshidi Michigan State University kordjams@msu.edu Abstract Tracking entities throughout a procedure de- scribed in a text is challenging due to the dy- namic nature of the world described in the pro- cess. Firstly, we propose to formulate this task as a question answering problem. This en- ables us to use pre-trained transformer-based language models on other QA benchmarks by adapting those to the procedural text un- derstanding. Secondly, since the transformer- based language models cannot encode the flow of events by themselves, we propose a Time- Stamped Language Model (TSLM model) to encode event information in LMs architec- ture by introducing the timestamp encoding. Our model evaluated on the Propara dataset shows improvements on the published state- of-the-art results with a 3.1% increase in F1 score. Moreover, our model yields better re- sults on the location prediction task on the NPN-Cooking dataset. This result indicates that our approach is effective for procedural text understanding in general. 1 Introduction A procedural text such as a recipe or an instruction usually describes the interaction between multiple entities and their attribute changes at each step of a process. For example, the photosynthesis pro- cedure can contain steps such as 1. Roots absorb water from soil; 2. The water flows to the leaf; 3. Light from the sun and CO2 enter the leaf; 4. The water, light, and CO2 combine into a mixture; 5. Mixture forms sugar. Procedural text under- standing is a machine reading comprehension task defined on procedural texts. Answering questions such as "what is the location of the mixture at step 4", in the above example, requires tracking enti- ties’ interactions to predict their attributes at each step (Dalvi et al., 2018; Bosselut et al., 2018). This is quite challenging due to the dynamic nature of the entities’ attributes in the context. Transformer-based language models have shown promising results on multi-hop or single-hop question answering benchmarks such as Hot- potQA (Yang et al., 2018), SQuAD (Rajpurkar et al., 2016), and Drop (Dua et al., 2019). However, it is hard to expect LMs to understand the flow of events and pay attention to the time in the proce- dure (e.g., step 4) without extra modeling efforts. In recent research, different approaches are taken to address procedural reasoning based on lan- guage models using QA formulations. Following the intuition that attributes of entities can be re- trieved based on the current and previous steps, DynaPro (Amini et al., 2020) modifies the input to only contain those sentences in the input at each time. This will provide a different input to the model based on each question to help it detect changes after adding each step. KG-MRC (Das et al., 2018) also generates a …

Content truncated to 500 words