How Knowledge Graph and Attention Help? A Qualitative Analysis into Bag-level Relation Extraction

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4662–4671 August 1–6, 2021. ©2021 Association for Computational Linguistics 4662 How Knowledge Graph and Attention Help? A Quantitative Analysis into Bag-level Relation Extraction Zikun Hu1 , Yixin Cao2 , Lifu Huang3 , Tat-Seng Chua1 1National University of Singapore 2S-Lab, Nanyang Technological University 3Computer Science Department, Virginia Tech zikunhu@nus.edu.sg, yixin.cao@ntu.edu.sg lifuh@vt.edu, chuats@comp.nus.edu.sg Abstract Knowledge Graph (KG) and attention mecha- nism have been demonstrated effective in in- troducing and selecting useful information for weakly supervised methods. However, only qualitative analysis and ablation study are pro- vided as evidence. In this paper, we contribute a dataset and propose a paradigm to quantita- tively evaluate the effect of attention and KG on bag-level relation extraction (RE). We find that (1) higher attention accuracy may lead to worse performance as it may harm the model’s ability to extract entity mention features; (2) the performance of attention is largely influ- enced by various noise distribution patterns, which is closely related to real-world datasets; (3) KG-enhanced attention indeed improves RE performance, while not through enhanced attention but by incorporating entity prior; and (4) attention mechanism may exacerbate the issue of insufficient training data. Based on these findings, we show that a straightfor- ward variant of RE model can achieve sig- nificant improvements (6% AUC on average) on two real-world datasets as compared with three state-of-the-art baselines. Our codes and datasets are available at https://github.com/zig- kwin-hu/how-KG-ATT-help. 1 Introduction Relation Extraction (RE) is crucial for Knowledge Graph (KG) construction and population. Most recent efforts rely on neural networks to learn effi- cient features from large-scale annotated data, thus correctly extract the relationship between entities. To save the manual annotation cost and alleviate the issue of data scarcity, distant supervision relation extraction (DSRE) (Mintz et al., 2009) is proposed and becomes increasingly popular as it can auto- matically generate large-scale labeled data. DSRE is based on a simple yet effective principle: if there is a relation between two entities in KG, then all sentences containing mentions of both entities are assumed to express this relation and will form a sentence bag as its annotations. Figure 1: Examples of disturbing bags in NYT-FB60K. Although effective, distant supervision may in- troduce noise to a sentence bag when the assump- tion fails — some sentences are not describing the target relation (Zeng et al., 2015) (a.k.a. noisy anno- tation). To alleviate the negative impacts of noise, recent studies (Lin et al., 2016; Ji et al., 2017; Du et al., 2018; Li et al., 2020) leveraged attention to select informative instances from a bag. Further- more, researchers introduced KG embeddings to enhance the attention mechanism (Hu et al., 2019; Han et al., 2018a). The basic idea is to utilize entity embeddings as the query to compute atten- tion scores, so that the sentences with high atten- tion weights are more likely to be valid annota- tions (Zhang et al., 2019). Previous …

Content truncated to 500 words