Label Verbalization and Entailment for Effective Zero and Few-Shot Relation Extraction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1199–1212 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 1199 Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction Oscar Sainz Oier Lopez de Lacalle Gorka Labaka Ander Barrena Eneko Agirre HiTZ Basque Center for Language Technologies - Ixa NLP Group University of the Basque Country (UPV/EHU) {oscar.sainz, oier.lopezdelacalle, gorka.labaka, ander.barrena, e.agirre}@ehu.eus Abstract Relation extraction systems require large amounts of labeled examples which are costly to annotate. In this work we reformulate re- lation extraction as an entailment task, with simple, hand-made, verbalizations of relations produced in less than 15 minutes per relation. The system relies on a pretrained textual entail- ment engine which is run as-is (no training ex- amples, zero-shot) or further fine-tuned on la- beled examples (few-shot or fully trained). In our experiments on TACRED we attain 63% F1 zero-shot, 69% with 16 examples per re- lation (17% points better than the best super- vised system on the same conditions), and only 4 points short of the state-of-the-art (which uses 20 times more training data). We also show that the performance can be improved significantly with larger entailment models, up to 12 points in zero-shot, giving the best re- sults to date on TACRED when fully trained. The analysis shows that our few-shot systems are especially effective when discriminating between relations, and that the performance difference in low data regimes comes mainly from identifying no-relation cases. 1 Introduction Given a context where two entities appear, the Rela- tion Extraction (RE) task aims to predict the seman- tic relation (if any) holding between the two entities. Methods that fine-tune large pretrained language models (LM) with large amounts of labelled data have established the state of the art (Yamada et al., 2020). Nevertheless, due to differing languages, domains and the cost of human annotation, there is typically a very small number of labelled exam- ples in real-world applications, and such models perform poorly (Schick and Schütze, 2021). As an alternative, methods that only need a few examples (few-shot) or no examples (zero-shot) have emerged. For instance, prompt based learning proposes hand-made or automatically learned task and label verbalizations (Puri and Catanzaro, 2019; Schick and Schütze, 2021; Schick and Schütze, 2020) as an alternative to standard fine-tuning (Gao et al., 2020; Scao and Rush, 2021). In these meth- ods, the prompts are input to the LM together with the example, and the language modelling objec- tive is used in learning and inference. In a dif- ferent direction, some authors reformulate the tar- get task (e.g. document classification) as a pivot task (typically question answering or textual entail- ment), which allows the use of readily available question answering (or entailment) training data (Yin et al., 2019; Levy et al., 2017). In all cases, the underlying idea is to cast the target task into a formulation which allows us to exploit the knowl- edge implicit in pre-trained LM (prompt-based) or general-purpose question answering or entailment engines (pivot …

Content truncated to 500 words