Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes

Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes Ofer Sabo1 Yanai Elazar1,2 Yoav Goldberg1,2 Ido Dagan1 1Computer Science Department, Bar Ilan University, Israel 2Allen Institute for Artificial Intelligence {ofersabo,yanaiela,yoav.goldberg,ido.k.dagan}@gmail.com Abstract We explore few-shot learning (FSL) for re- lation classification (RC). Focusing on the realistic scenario of FSL, in which a test instance might not belong to any of the target categories (none-of-the-above, [NOTA]), we first revisit the recent popular dataset struc- ture for FSL, pointing out its unrealistic data distribution. To remedy this, we propose a novel methodology for deriving more realistic few-shot test data from available datasets for supervised RC, and apply it to the TACRED dataset. This yields a new challenging bench- mark for FSL-RC, on which state of the art models show poor performance. Next, we ana- lyze classification schemes within the popular embedding-based nearest-neighbor approach for FSL, with respect to constraints they im- pose on the embedding space. Triggered by this analysis, we propose a novel classifica- tion scheme in which the NOTA category is represented as learned vectors, shown empiri- cally to be an appealing option for FSL. 1 Introduction We consider relation classification—an important sub-task of relation extraction—in which one is interested in determining, given a text with two marked entities, whether the entities conform to one of pre-determined relations, or not. While supervised methods for this task exist and work relatively well (Baldini Soares et al., 2019; Zhang et al., 2018; Wang et al., 2016; Miwa and Bansal, 2016), they require large amounts of training data, which is hard to obtain in practice. We are therefore interested in a data-lean sce- nario in which users provide only a handful of training examples for each relation they are in- terested in. This has been formalized in the ma- chine learning community as few-shot learning (FSL) (§2). FSL for RC has been recently addressed by the work of Han et al. (2018) and Gao et al. (2019), who introduced the FewRel 1.0 and shortly after the FewRel 2.0 challenges, in which researchers are provided with a large labeled dataset of back- ground relations, and are tasked with producing strong few-shot classifiers: classifiers that will work well given a few labeled examples of rela- tions not seen in the training set. The task became popular, with scores on FewRel 1.0 achieving an accuracy of 93.9% (Baldini Soares et al., 2019), surpassing the human level performance of 92.2%. Results on FewRel 2.0 are lower, at 80.3% for the best system (Gao et al., 2019), but are still very high considering the difficulty of the task. Is few-shot relation classification solved? We show that this is far from being the case. We argue that the evaluation protocol in FewRel 1.0 is based on highly unrealistic assumptions on how the mod- els will be used in practice, and while FewRel 2.0 tried to amend it, its evaluation setup remains highly unrealistic (§3.1). Therefore, we propose a methodology to transform supervised datasets into corresponding realistic few-shot evaluation sce- narios …

Content truncated to 500 words