ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3470–3479 June 6–11, 2021. ©2021 Association for Computational Linguistics 3470 ZS-BERT: Towards Zero-Shot Relation Extraction with Attribute Representation Learning Chih-Yao Chen Progam of Data Science National Taiwan University Taipei, Taiwan r09946001@ntu.edu.tw Cheng-Te Li Institute of Data Science National Cheng Kung University Tainan, Taiwan chengte@ncku.edu.tw Abstract While relation extraction is an essential task in knowledge acquisition and representation, and new-generated relations are common in the real world, less effort is made to predict unseen relations that cannot be observed at the train- ing stage. In this paper, we formulate the zero- shot relation extraction problem by incorporat- ing the text description of seen and unseen re- lations. We propose a novel multi-task learn- ing model, zero-shot BERT (ZS-BERT), to di- rectly predict unseen relations without hand- crafted attribute labeling and multiple pairwise classifications. Given training instances con- sisting of input sentences and the descriptions of their relations, ZS-BERT learns two func- tions that project sentences and relation de- scriptions into an embedding space by jointly minimizing the distances between them and classifying seen relations. By generating the embeddings of unseen relations and new- coming sentences based on such two functions, we use nearest neighbor search to obtain the prediction of unseen relations. Experiments conducted on two well-known datasets exhibit that ZS-BERT can outperform existing meth- ods by at least 13.54% improvement on F1 score. 1 Introduction Relation extraction is an important task in the natu- ral language processing field, which aims to infer the semantic relation between a pair of entities within a given sentence. There are many appli- cations based on relation extraction, such as ex- tending knowledge bases (KB) (Lin et al., 2015) and improving question answering task (Xu et al., 2016). Existing approaches to this task usually require large-scale labeled data. However, the la- beling cost is a considerable difficulty. Some re- cent studies generate labeled data based on distant supervision (Mintz et al., 2009; Ji et al., 2017). Nevertheless, when putting the relation extraction task in the wild, existing supervised models cannot well recognize the relations of instances that are extremely rare or even never covered by the train- ing data. That said, in the real-world setting, we should not presume the relations/classes of new- coming sentences are always included in the train- ing data. Thus it is crucial to invent new models to predict new classes that are not defined or ob- served beforehand. Such a task is referred as zero- shot learning (ZSL) (Norouzi et al., 2013; Lampert et al., 2014; Ba et al., 2015; Kodirov et al., 2017). The idea of ZSL is to connect seen and the un- seen classes by finding an intermediate semantic representation. Unlike the common way to train a supervised model, seen and unseen classes are disjoint at training and testing stages. Hence, ZSL models need to generate transferable knowledge between them. With a model for ZSL relation ex- …

Content truncated to 500 words