Exploring the zero-shot limit of FewRel

Proceedings of the 28th International Conference on Computational Linguistics, pages 1447–1451 Barcelona, Spain (Online), December 8-13, 2020 1447 Exploring the zero-shot limit of FewRel Alberto Cetoli QBE Europe 30 Fenchurch St, Billingsgate, London EC3M 3BD alberto.cetoli@uk.qbe.com Abstract This paper proposes a general purpose relation extractor (RE) that uses Wikidata descriptions to represent the relation’s surface form. The results are tested on the FewRel 1.0 dataset, which provides an excellent framework for training and evaluating the proposed zero-shot learning sys- tem in English. This relation extractor architecture exploits the implicit knowledge of a language model through a question-answering approach1. 1 Introduction and related works The FewRel dataset (Han et al., 2018) has been tailored for the task of few shot learning, where the model is presented with a limited sample of candidates in English for each relation (Table 1). The evaluation is performed on a set of relations unseen during training, thus forcing the system to abstract and generalize some core language features from the examples. A plethora of methods has flourished to address the challenge posed by the FewRel set (Ye and Ling, 2019; Gao et al., 2019), eventually achieving super- human performance in the work of (Baldini Soares et al., 2019). Understandably, these methods greatly benefit from recent progress on pre-trained language models like BERT (Devlin et al., 2018), leveraging on the model’s implicit semantic knowledge. Building extractors with a broad purpose is of utmost importance in information extraction, both at a theoretical and application level. In this spirit, the zero- shot learning approach implements the scenario where the model is never presented with any examples. An early implementation of this technique can be found in the open information extraction framework (Banko et al., 2007; Fader et al., 2011) which represents the relations with their surface forms. More recently the work by (Levy et al., 2017) builds on Question-Answering techniques to build new datasets and models for RE: The relations are represented as questions and the answers are the connected entities. Another work by (Obamuyide and Vlachos, 2018) uses Textual Entailment for relation extraction. Since many sentences can express the same relation, typically zero-shot relation extractors are limited in their ability to generalize and do not perform as well as few-shot learning models. This work aims to improve upon prior systems by leveraging on the advancements of question an- swering models and by representing relations using their surface forms as they appear on Wikidata. The FewRel dataset is an ideal playground for this task: It forces the system to generalize by evaluating on unseen relations while at the same time mapping every relation to a Wikidata identifier. The contributions of this paper are as in the following • Exploring the limitations and establishing a benchmark for zero-shot relation extraction on FewRel 1.0. The outcome shows how close zero-shot RE can be to one-shot learning. • Introducing a new technique to exploit the implicit knowledge of a language model fine-tuned on SQUAD. • Building a general purpose system that only …

Content truncated to 500 words