Revisiting Relation Extraction in the era of Large Language Models

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 15566–15589 July 9-14, 2023 ©2023 Association for Computational Linguistics Revisiting Relation Extraction in the era of Large Language Models Somin Wadhwa Silvio Amir Northeastern UniversityByron C. Wallace {wadhwa.s, s.amir, b.wallace}@northeastern.edu Abstract Relation extraction (RE) is the core NLP task of inferring semantic relationships between en- tities from text. Standard supervised RE tech- niques entail training modules to tag tokens comprising entity spans and then predict the relationship between them. Recent work has instead treated the problem as a sequence-to- sequence task, linearizing relations between entities as target strings to be generated condi- tioned on the input. Here we push the limits of this approach, using larger language models (GPT-3 and Flan-T5 large) than considered in prior work and evaluating their performance on standard RE tasks under varying levels of supervision. We address issues inherent to eval- uating generative approaches to RE by doing human evaluations, in lieu of relying on exact matching. Under this refined evaluation, we find that: (1) Few-shot prompting with GPT-3 achieves near SOTA performance, i.e., roughly equivalent to existing fully supervised mod- els; (2) Flan-T5 is not as capable in the few- shot setting, but supervising and fine-tuning it with Chain-of-Thought (CoT) style explana- tions (generated via GPT-3) yields SOTA re- sults. We release this model as a new baseline for RE tasks1. 1 Introduction Relation extraction (RE) is the task of identifying entities and their semantic relationships from texts. Standard supervised approaches (Eberts and Ulges, 2019a) to RE learn to tag entity spans and then classify relationships (if any) between these. More recent work has shown that conditional language models can capably perform this task—achieving SOTA or near-SOTA results—when trained to out- put linearized strings encoding entity pairs and their relations (Paolini et al., 2021; Lu et al., 2022b; Huguet Cabot and Navigli, 2021). However, to date such work has considered only moderately sized 1https://sominw.com/ACL23LLMs REBEL (Baseline) Model Performance on CoNLL Micro-F1 Score Few-Shot GPT-3 Few-Shot GPT-3 + CoT Fine-Tuned Flan-T5 Fine-Tuned Flan-T5 + CoT 1 2 3 4 Figure 1: RE performance of LLMs on the CoNLL dataset. 1 Few-shot GPT-3 slightly outperforms the ex- isting fully supervised SOTA method (Huguet Cabot and Navigli 2021; dotted horizontal line). 2 Eliciting CoT reasoning from GPT-3 further improves few-shot per- formance. 3 Fine-tuning Flan-T5 (large) is competitive with, but no better than, existing supervised methods, but 4 supervising Flan-T5 with CoT reasoning elicited from GPT-3 substantially outperforms all other models. pre-trained models for RE such as BART (Paolini et al., 2021; Huguet Cabot and Navigli, 2021). In this work we investigate the use of very large language models—-including GPT-3 (Brown et al., 2020b)—for end-to-end relation extraction via gen- eration. Our contributions are as follows. 1. We show that few-shot learning with GPT-3 yields near SOTA performance on standard RE datasets, outperforming fully supervised models. 2. We find that Flan-T5 (large; Chung et al. 2022) is not as capable, even …

Content truncated to 500 words