SLICE: Supersense-based Lightweight Interpretable Contextual Embeddings

Proceedings of the 28th International Conference on Computational Linguistics, pages 328–338 Barcelona, Spain (Online), December 8-13, 2020 328 Heterogeneous Graph Neural Networks to Predict What Happen Next Jianming Zheng, Fei Cai∗, Yangxiang Ling, Honghui Chen Science and Technology on Information Systems Engineering Laboratory National University of Defense Technology Changsha, China {zhengjianming12,caifei,lingyanxiang,chenhonghui}@nudt.edu.cn Abstract Given an incomplete event chain, script learning aims to predict the missing event, which can support a series of NLP applications. Existing work cannot well represent the heterogeneous relations and capture the discontinuous event segments that are common in the event chain. To address these issues, we introduce a heterogeneous-event (HeterEvent) graph network. In partic- ular, we employ each unique word and individual event as nodes in the graph, and explore three kinds of edges based on realistic relations (e.g., the relations of word-and-word, word-and-event, event-and-event). We also design a message passing process to realize information interactions among homo or heterogeneous nodes. And the discontinuous event segments could be explicitly modeled by finding the specific path between corresponding nodes in the graph. The experi- mental results on one-step and multi-step inference tasks demonstrate that our ensemble model HeterEvent[W+E] can outperform existing baselines. 1 Introduction Event chain, also known as script (Roger and Robert, 1977), is a structural knowledge format that mod- els stereotypical human activities in a given scenario, e.g., “dining at a restaurant” and “catching a thief” in Fig. 1. Representing such knowledge in a machine-readable way can help machine understand the semantics of natural language and further perform human-like inferences. Besides, event representation can also support a series of downstream applications, such as question answering (Li et al., 2019), dis- course understanding (Huang et al., 2019) and information extraction (Liu et al., 2019a; Zheng et al., 2019), text classification (Zheng et al., 2020b), etc. Existing work on event representation mainly model event chain from three aspects, the intra-event based (Weber et al., 2018; Granroth-Wilding and Clark, 2016), the individual-event based (Li et al., 2018; Wang et al., 2017) and the event-segment (Lv et al., 2019) based models. These methods concen- trate on depicting the relations among homogeneous modeling objectives, e.g., the inter-event relations. However, the relations among heterogeneous ones, e.g., the subordinated relations between word and event, which are also critical to the event chain, have not yet been taken into account in previous work. Besides, (Lv et al., 2019) found the event segments, a set of individual events related to each other, were helpful to predict the missing event and event segments could be continuous or discontinuous (See Fig. 1). Although the self-attention mechanism can implicitly represent such discontinuous event segments by greedily making connections among all events (Lv et al., 2019), it inevitably introduces noises. In this paper, we attempt to deal with these issues by proposing a heterogeneous-event (HeterEvent) graph network . Specifically, we define two different types of nodes in the HeterEvent graph, includ- ing word and event nodes, which respectively represent unique words and individual events in the event chain. Then …

Content truncated to 500 words