Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6947 - 6958 December 7-11, 2022 ©2022 Association for Computational Linguistics Cross-stitching Text and Knowledge Graph Encoders for Distantly Supervised Relation Extraction Qin Dai∗1, Benjamin Heinzerling∗1,2, Kentaro Inui1,2 1Tohoku University 2RIKEN AIP qin.dai.b8@tohoku.ac.jp, benjamin.heinzerling@riken.jp kentaro.inui@tohoku.ac.jp Abstract Bi-encoder architectures for distantly- supervised relation extraction are designed to make use of the complementary information found in text and knowledge graphs (KG). However, current architectures suffer from two drawbacks. They either do not allow any sharing between the text encoder and the KG encoder at all, or, in case of models with KG-to-text attention, only share infor- mation in one direction. Here, we introduce cross-stitch bi-encoders, which allow full interaction between the text encoder and the KG encoder via a cross-stitch mechanism. The cross-stitch mechanism allows sharing and updating representations between the two encoders at any layer, with the amount of sharing being dynamically controlled via cross-attention-based gates. Experimental results on two relation extraction benchmarks from two different domains show that enabling full interaction between the two encoders yields strong improvements. https://github.com/cl-tohoku/xbe 1 Introduction Identifying semantic relations between textual men- tions of entities is a key task for information extrac- tion systems. For example, consider the sentence: (1) Aspirin is widely used for short-term treat- ment of pain, fever or colds. Assuming an inventory of relations such as may_treat or founded_by, a relation extraction (RE) system should recognize the predicate in (1) as an instance of a may_treat relation and ex- tract a knowledge graph (KG) triple like (ASPIRIN, may_treat, PAIN). RE systems are commonly trained on data obtained via Distant Supervision (DS, Mintz et al., 2009): Given a KG triple, i.e., a pair of entities and a relation, one assumes that ∗ Equal contribution all sentences mentioning both entities express the relation and collects all such sentences as positive examples. DS allows collecting large amounts of training data, but its assumption is often violated: (2) The tumor was remarkably large in size, and pain unrelieved by aspirin. (3) Elon Musk fired some SpaceX employees who were talking smack about ... Sentence (2) is a false positive example of a may_treat relation since it describes a failed treat- ment. Sentence (3) is, strictly speaking, a false pos- itive of a founded_by relation since this sentence is not about founding companies, but can also be seen as indirect evidence, since founders are often in a position that allows them to fire employees. We refer to false positive and indirectly relevant examples like (2) and (3) as noisy sentences. A common approach for dealing with noisy sen- tences is to use the KG as a complementary source of information. Models taking this approach are typically implemented as bi-encoders, with one en- coder for textual input and one encoder for KG input. They are trained to rely more on the text en- coder when given informative sentences and more on the KG encoder when faced with noisy ones (Weston et al., 2013; Han …

Content truncated to 500 words