FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6250–6255, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6250 FewRel 2.0: Towards More Challenging Few-Shot Relation Classification Tianyu Gao1, Xu Han1, Hao Zhu1,2, Zhiyuan Liu1∗, Peng Li3, Maosong Sun1, Jie Zhou3 1Department of Computer Science and Technology, Tsinghua University, Beijing, China 2Carnegie Mellon University, Pittsburgh, PA, USA 3Pattern Recognition Center, WeChat AI, Tencent Inc, China {gty16,hanxu17}@mails.tsinghua.edu.cn,zhuhao@cmu.edu liuzy@tsinghua.edu.cn,patrickpli@tencent.com sms@tsinghua.edu.cn,withtomzhou@tencent.com Abstract We present FewRel 2.0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a hand- ful of instances? (2) Can they detect none- of-the-above (NOTA) relations? To construct FewRel 2.0, we build upon the FewRel dataset (Han et al., 2018) by adding a new test set in a quite different domain, and a NOTA re- lation choice. With the new dataset and ex- tensive experimental analysis, we found (1) that the state-of-the-art few-shot relation clas- sification models struggle on these two as- pects, and (2) that the commonly-used tech- niques for domain adaptation and NOTA de- tection still cannot handle the two challenges well. Our research calls for more attention and further efforts to these two real-world issues. All details and resources about the dataset and baselines are released at https: //github.com/thunlp/fewrel. 1 Introduction Few-shot learning, which requires models to han- dle new classification tasks with only a handful of training instances, has drawn much attention in recent years (Ravi and Larochelle, 2017; Vinyals et al., 2016; Munkhdalai and Yu, 2017; Snell et al., 2017). To advance this field in NLP, Han et al. (2018) propose FewRel, a large-scale dataset to explore few-shot learning in relation classification. Many efforts (Gao et al., 2019; Soares et al., 2019) have been devoted to the new task and some of the methods even exceed human performance1 on FewRel. Based on the dataset FewRel, we pro- pose FewRel 2.0, a new task containing two real- world issues that FewRel ignores: (1) few-shot domain adaptation, and (2) few-shot none-of-the- above detection. ∗ Corresponding author 1https://thunlp.github.io/fewrel.html 5-Way 1-Shot 5-Way 5-Shot 10-Way 1-Shot 10-Way 5-Shot 0 20 40 60 80 100 Accuracy (%) Results on Few-Shot DA Best on FewRel Best on Few-Shot DA 5-Way 1-Shot 5-Way 5-Shot 0 20 40 60 80 100 Accuracy (%) Results on Few-Shot NOTA Best on FewRel Best on 50% Few-Shot NOTA Figure 1: The comparison between the best results of the current models on FewRel, few-shot DA and few- shot NOTA. From the figures we can see that even the state-of-the-art models struggle on the new tasks. Few-shot domain adaptation (few-shot DA) aims to evaluate the abilities of few-shot models to transfer across domains, which is crucial for real- world applications, since the test domains usually lack of annotations and could differ vastly from the training domains. To this end, we construct a new test set sharing great disparities with …

Content truncated to 500 words