A Graph Enhanced BERT Model for Event Prediction

A Graph Enhanced BERT Model for Event Prediction Li Du †, Xiao Ding † ∗, Yue Zhang ‡, Kai Xiong †, Ting Liu †, and Bing Qin † † Research Center for Social Computing and Information Retrieval Harbin Institute of Technology, China {ldu, xding, kxiong, tliu,qinb}@ir.hit.edu.cn ‡ School of Engineering, Westlake University zhangyue@westlake.edu.cn Abstract Predicting the subsequent event for an exist- ing event context is an important but challeng- ing task, as it requires understanding the un- derlying relationship between events. Previ- ous methods propose to retrieve relational fea- tures from event graph to enhance the model- ing of event correlation. However, the spar- sity of event graph may restrict the acquisition of relevant graph information, and hence influ- ence the model performance. To address this issue, we consider automatically building of event graph using a BERT model. To this end, we incorporate an additional structured vari- able into BERT to learn to predict the event connections in the training process. Hence, in the test process, the connection relationship for unseen events can be predicted by the struc- tured variable. Results on two event prediction tasks: script event prediction and story ending prediction, show that our approach can outper- form state-of-the-art baseline methods. 1 Introduction Understanding the semantics of events and their un- derlying connections is a long-standing task in nat- ural language processing (Minsky, 1974; Schank, 1975). Much research has been done on extracting script knowledge from narrative texts, and mak- ing use of such knowledge for predicting a likely subsequent event given a set of context events. A key issue to fulfilling such tasks is the mod- eling of event relation information. To this end, early work exploited event pair relations (Cham- bers, 2008; Jans et al., 2012; Granroth and Clark, 2016) and temporal information (Pichotta, 2016; Pichotta and Mooney, 2016). The former has been used for event prediction by using embedding methods, where the similarity between subsequent events and context events are measured and used for candidate ranking. The latter has been used ∗Corresponding author Figure 1: (a) An example for event prediction. (b) Given an event sequence, retrieval-based methods lookup structural information of events from event graph. However, in the test process, part of events may be not covered by the event graph, hence their connection information is unavailable. Different from retrieval-based methods, GraphBERT is able to predict the connection strength between events. for neural network methods, where models such as LSTMs have been used to model a chain of context events. There has also been work integrating the two methods (Wang et al., 2017). Despite achieving certain effectiveness, the above methods do not fully model the underlying connection between context events. As shown in Figure 1 (a), given the facts that Jason had been overstretched at work, He decided to change job and Jason finds a new job, the subsequent event Ja- son is satisfied with his new job is more likely than Jason feels much stressed at his new job, which can be inferred …

Content truncated to 500 words