Towards Realistic Few-Shot Relation Extraction

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5338–5345 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5338 Towards Realistic Few-Shot Relation Extraction Sam Brody, Sichao Wu, Adrian Benton Bloomberg 731 Lexington Ave New York, NY 10022 USA {sbrody18,swu389,abenton10}@bloomberg.net Abstract In recent years, few-shot models have been ap- plied successfully to a variety of NLP tasks. Han et al. (2018) introduced a few-shot learn- ing framework for relation classification, and since then, several models have surpassed hu- man performance on this task, leading to the impression that few-shot relation classification is solved. In this paper we take a deeper look at the efficacy of strong few-shot classification models in the more common relation extrac- tion setting, and show that typical few-shot evaluation metrics obscure a wide variability in performance across relations. In particular, we find that state of the art few-shot relation classification models overly rely on entity type information, and propose modifications to the training routine to encourage models to better discriminate between relations involving simi- lar entity types. 1 Introduction Few-shot approaches have been explored in a va- riety of natural language processing (NLP) tasks, such as machine translation (Gu et al., 2018) and textual entailment (Yin et al., 2020), as well as an assortment of classification and natural language in- ference tasks (Yan et al., 2018; Brown et al., 2020b). The introduction of large, pretrained transformer language models in NLP increased the promise of building systems that can perform complex NLP tasks from only a small number of training exam- ples (Brown et al., 2020a). Han et al. (2018) intro- duced a few-shot learning framework for relation classification (FewRel), and recently several sys- tems have achieved near-human performance on this task – in some settings even exceeding it.1 These stunning results might give the impression that FewRel has been solved, and that these sys- tems can be used to extract any relation of inter- est from a collection of text (e.g., to populate a 1thunlp.github.io/1/fewrel1.html knowledge-base from web documents) with only a few example instances. In this paper, we take a deeper look at the ap- plicability of high performing few-shot relation classification models in a more realistic relation extraction (RE) setting. We find that although transformer-based models achieve high accuracy at the FewRel task, this obscures a wide variability in performance across relation types. In particular, we find that state-of-the-art FewRel models heavily rely on entity type information, and thus are unable to discriminate between many types of relations that are trivial for humans (e.g., spouse-of vs. child- of). However, we find that enriching the training data with relations with similar entity types forces the model to attend less to entity type information, and in a ranking evaluation, improves performance on unseen relations by up to 24% precision at 50, absolute. 2 Few-shot Relation Extraction 2.1 FewRel 1.0 The FewRel challenge (Han et al., 2018) intro- duced the few-shot paradigm to relation classifi- cation. The authors provided a dataset and eval- …

Content truncated to 500 words