Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking

Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking Yuanhe Tian♥, Yan Song♠†, Fei Xia♥ ♥University of Washington ♠The Chinese University of Hong Kong (Shenzhen) ♥{yhtian, fxia}@uw.edu ♠songyan@cuhk.edu.cn Abstract Relation extraction (RE) is an important natu- ral language processing task that predicts the relation between two given entities, where a good understanding of the contextual infor- mation is essential to achieve an outstanding model performance. Among different types of contextual information, the auto-generated syntactic information (namely, word depen- dencies) has shown its effectiveness for the task. However, most existing studies require modifications to the existing baseline archi- tectures (e.g., adding new components, such as GCN, on the top of an encoder) to lever- age the syntactic information. To offer an alternative solution, we propose to leverage syntactic information to improve RE by train- ing a syntax-induced encoder on auto-parsed data through dependency masking. Specifi- cally, the syntax-induced encoder is trained by recovering the masked dependency connec- tions and types in first, second, and third or- ders, which significantly differs from existing studies that train language models or word embeddings by predicting the context words along the dependency paths. Experimental results on two English benchmark datasets, namely, ACE2005EN and SemEval 2010 Task 8 datasets, demonstrate the effectiveness of our approach for RE, where our approach outper- forms strong baselines and achieve state-of- the-art results on both datasets.1 1 Introduction Relation extraction (RE) provides deep analyses of the input text by extracting the relation between two given entities in the input. Therefore, it is an im- portant task in natural language processing (NLP) and is widely used in many downstream NLP appli- cations such as summarization (Wang and Cardie, 2012), question answering systems (Xu et al., 2016) †Corresponding author. 1The code involved in this paper are released at https: //github.com/synlp/RE-DMP. and text mining (Distiawan et al., 2019). To cor- rectly extract the relation between two entities, it normally requires a good modeling and analysis of the input text. Recent models such as LSTM, Trans- formers (Vaswani et al., 2017), and pre-trained lan- guage models (e.g., BERT (Devlin et al., 2019) and XLNet (Yang et al., 2019)) have significantly improved the performance of RE models with an important reason of their encoding power on con- textual information. However, such models still reach a bottleneck because it is hard for them to capture structural information of the running text (which is essential for RE) by modeling the text as a linear sequence of words. To deal with this situation, extra knowledge and features (e.g., syn- tactic knowledge) are used in many studies, while of all choices, the dependency parses have been widely used and demonstrated to be effective (Xu et al., 2015; Zhang et al., 2018; Guo et al., 2019; Mandya et al., 2020; Sun et al., 2020; Yu et al., 2020b; Tian et al., 2021), for the reason that the dependency trees are able to provide long-distance word-word relations which are important structural complement to existing models for RE. To leverage dependency information, …

Content truncated to 500 words