EventBERT: Incorporating Event-based Semantics for Natural Language Understanding

EventBERT: Incorporating Event-based Semantics for Natural Language Understanding Anni Zou1,2,3, Zhuosheng Zhang1,2,3, Hai Zhao1,2,3,∗ 1 Department of Computer Science and Engineering, Shanghai Jiao Tong University 2 Key Laboratory of Shanghai Education Commission for Intelligent Interaction and Cognitive Engineering, Shanghai Jiao Tong University 3MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University {annie0103,zhangzs}@sjtu.edu.cn,zhaohai@cs.sjtu.edu.cn Abstract Natural language understanding tasks require a comprehensive understanding of natural language and further reasoning about it, on the basis of holistic information at different levels to gain com- prehensive knowledge. In recent years, pre-trained language models (PrLMs) have shown im- pressive performance in natural language understanding. However, they rely mainly on extracting context-sensitive statistical patterns without explicitly modeling linguistic information, such as semantic relationships entailed in natural language. In this work, we propose EventBERT, an event-based semantic representation model that takes BERT as the backbone and refines with event-based structural semantics in terms of graph convolution networks. EventBERT benefits simultaneously from rich event-based structures embodied in the graph and contextual semantics learned in pre-trained model BERT. Experimental results on the GLUE benchmark show that the proposed model consistently outperforms the baseline model. 1 Introduction Recent years have witnessed deep pre-trained language models (PrLM) such as ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), XLNet (Yang et al., 2019) and ERNIE (Sun et al., 2020) signifi- cantly prospering the performance of a wide range of natural language understanding (NLU) tasks. The remarkable advancements brought by PrLM have shown the effectiveness of leveraging contextualized representation. However, they mainly rest on extracting context-sensitive statistical patterns without explicitly modeling linguistic information such as semantic relationships in natural language. It is clear that natural language itself abounds with ample, multi-level linguistic information. Although PrLMs like BERT implicitly represent linguistic knowledge more or less (Rogers et al., 2020), studies disclose that linguistic knowledge is far from fully absorbed (Ettinger, 2020; Rogers et al., 2020). There- fore, there emerges a series of derivatives of PrLM intending to fuse explicit linguistic knowledge so as to acquire better language representation, including syntactic (Bai et al., 2021; Xu et al., 2021; Zhang et al., 2020b) and semantic information (Zhang et al., 2020a; Guo et al., 2020b; Guan et al., 2021). In cognition practice, human needs to distill semantics of different levels to gain a comprehensive understanding, whereas neural language models learn semantic representation to deal with downstream tasks (Geeraerts and Cuyckens, 2007). Thus, effective learning of semantic knowledge plays a crucial role in NLU tasks and has gained growing attention recently. For instance, Zhang et al. (2020a) proposed SemBERT, which directly connects multiple predicate-argument structures acquired by semantic role labeler (SRL) to get the joint representation. The essence of SRL (Shi and Lin, 2019) lies in that every sentence possesses multiple predicate- specific structures which can represent different frames of events, while semantic roles express the ab- stract role that arguments of a predicate can take in the event. Besides, the events inside a sentence have interactions with each other that serve …

Content truncated to 500 words