Sentence Embedding Alignment for Lifelong Relation Extraction

Proceedings of NAACL-HLT 2019, pages 796–806 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 796 Sentence Embedding Alignment for Lifelong Relation Extraction Hong Wang†, Wenhan Xiong†, Mo Yu‡∗, Xiaoxiao Guo‡∗, Shiyu Chang‡, William Yang Wang† † University of California, Santa Barbara ‡ IBM Research {hongwang600, xwhan, william}@cs.ucsb.edu, yum@us.ibm.com, {xiaoxiao.guo, shiyu.chang}@ibm.com Abstract Conventional approaches to relation extraction usually require a fixed set of pre-defined re- lations. Such requirement is hard to meet in many real applications, especially when new data and relations are emerging incessantly and it is computationally expensive to store all data and re-train the whole model every time new data and relations come in. We for- mulate such a challenging problem as lifelong relation extraction and investigate memory- efficient incremental learning methods without catastrophically forgetting knowledge learned from previous tasks. We first investigate a modified version of the stochastic gradient methods with a replay memory, which surpris- ingly outperforms recent state-of-the-art life- long learning methods. We further propose to improve this approach to alleviate the forget- ting problem by anchoring the sentence em- bedding space. Specifically, we utilize an ex- plicit alignment model to mitigate the sentence embedding distortion of the learned model when training on new data and new relations. Experiment results on multiple benchmarks show that our proposed method significantly outperforms the state-of-the-art lifelong learn- ing approaches. 1 Introduction The task of relation detection/extraction aims to recognize entity pairs’ relationship from given contexts. As an essential component for structured information extraction, it has been widely used in downstream tasks such as automatic knowledge- based completion (Riedel et al., 2013) and ques- tion answering (Yih et al., 2015; Yu et al., 2017). Existing relation detection methods always as- sume a closed set of relations and perform once- ∗ Co-mentoring Code and dataset can be found in this repository: https://github.com/hongwang600/Lifelong_ Relation_Detection and-for-all training on a fixed dataset. While making the evaluation straightforward, this set- ting clearly limits the usage of these methods in realistic applications, where new relations keep emerging over time. To build an evolving system which automatically keeps up with the dynamic data, we consider a more practical lifelong learn- ing setting (also called continual learning) (Ring, 1994; Thrun, 1998; Thrun and Pratt, 2012), where a learning agent learns from a sequence of tasks, where each of them includes a different set of re- lations. In such scenarios, it is often infeasible to combine the new data with all previous data and re-train the model using the combined dataset, es- pecially when the training set for each task is huge. To enable efficient learning in such scenar- ios, recent lifelong learning research (Kirkpatrick et al., 2016; Lopez-Paz and Ranzato, 2017) pro- pose to learn the tasks incrementally, while at the same time preventing catastrophic forgetting (Mc- Closkey and Cohen, 1989; Ratcliff, 1990; McClel- land et al., 1995; French, 1999), i.e., the model abruptly forgets knowledge learned on previous tasks when learning on the new task. Current life- long learning approaches address …

Content truncated to 500 words