KnowSemLM: A Knowledge Infused Semantic Language Model

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 550–562 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 550 KnowSemLM : A Knowledge Infused Semantic Language Model Haoruo Peng1, Qiang Ning1, Dan Roth1,2 Department of Computer Science 1University of Illinois at Urbana-Champaign, Urbana, IL 61801, USA 2University of Pennsylvania, Philadelphia, PA 19104, USA {hpeng7,qning2}@illinois.edu, danroth@seas.upenn.edu Abstract Story understanding requires developing ex- pectations of what events come next in text. Prior knowledge – both statistical and declar- ative – is essential in guiding such expecta- tions. While existing semantic language mod- els (SemLM) capture event co-occurrence in- formation by modeling event sequences as se- mantic frames, entities, and other semantic units, this paper aims at augmenting them with causal knowledge (i.e., one event is likely to lead to another). Such knowledge is mod- eled at the frame and entity level, and can be obtained either statistically from text or stated declaratively. The proposed method, KnowSemLM1, infuses this knowledge into a semantic LM by joint training and inference, and is shown to be effective on both the event cloze test and story/referent prediction tasks. 1 Introduction Natural language understanding requires a coher- ent understanding of a series of events or actions in a story. In story comprehension, we need to under- stand not only what events have appeared in text, but also what is likely to happen next. While event extraction has been well studied (Ji and Grishman, 2008; Huang and Riloff, 2012; Li et al., 2013; Peng et al., 2016; Nguyen et al., 2016; Nguyen and Grishman, 2016), the task of predicting fu- ture events (Radinsky et al., 2012; Radinsky and Horvitz, 2013) has received less attention. One perspective is to utilize the co-occurrence information between past and future events learned from a large corpus, which has been stud- ied in script learning works (Chambers and Ju- rafsky, 2008; Pichotta and Mooney, 2014, 2016a; Peng and Roth, 2016; Peng et al., 2017). However, only considering co-occurrence information is not 1Related resources refer to https://cogcomp. seas.upenn.edu/page/publication_view/886. sufficient for modeling event sequences in natural language. Human decisions on the likelihood of a specific event depend on both local context – what has happened earlier in text – and global context – knowledge gained from human experience. This paper leverages both the local and global context information to model event sequences, and shows that it can lead to more accurate predictions of fu- ture events. For example, the following text snip- pet describes a scenario of someone taking a flight: ... I checked in at the counter, took my luggage to the security area, got cleared ten minutes in ad- vance, and waited for my plane ... This example consists of a series of events, i.e., “check in (a flight)”, “be cleared (at the secu- rity)”, “wait for (the plane)”, etc., which humans who have traveled by plane are very familiar with. However, this event sequence appears infrequently in text.2 Consequently, only relying on event co- occurrence in text …

Content truncated to 500 words