Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8229–8239, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8229 Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start Wenpeng Yin1, Nazneen Fatema Rajani1, Dragomir Radev1,2, Richard Socher1 and Caiming Xiong1 1Salesforce Research and 2Yale University wyin@salesforce.com Abstract A standard way to address different NLP prob- lems is by first constructing a problem-specific dataset, then building a model to fit this dataset. To build the ultimate artificial intelligence, we desire a single machine that can handle diverse new problems, for which task-specific annota- tions are limited. We bring up textual entail- ment as a unified solver for such NLP prob- lems. However, current research of textual en- tailment has not spilled much ink on the fol- lowing questions: (i) How well does a pre- trained textual entailment system generalize across domains with only a handful of domain- specific examples? and (ii) When is it worth transforming an NLP task into textual entail- ment? We argue that the transforming is un- necessary if we can obtain rich annotations for this task. Textual entailment really matters par- ticularly when the target NLP task has insuffi- cient annotations. Universal NLP1 can be probably achieved through different routines. In this work, we introduce Universal Few-shot textual En- tailment (UFO-ENTAIL). We demonstrate that this framework enables a pretrained en- tailment model to work well on new en- tailment domains in a few-shot setting, and show its effectiveness as a unified solver for several downstream NLP tasks such as question answering and coreference res- olution when the end-task annotations are limited. Code: https://github.com/ salesforce/UniversalFewShotNLP 1 Introduction Nowadays, the whole NLP journey has been broken down into innumerable sub-tasks. We often solve 1“Universal NLP” here means using a single machine to address diverse NLP problems. This is different from using the same machine learning algorithm such as convolution nets to solve tasks because the latter still results in task-specific models which can not solve other tasks. each task separately by first gathering task-specific training data and then tuning a machine learning system to learn the patterns in the data. Constrained by the current techniques, the journey has to be performed in this way. By a forward-looking per- spective, instead, a single machine that can handle diverse (seen and unseen) tasks is desired. The rea- son is that we cannot always rely on expensive hu- man resources to annotate large-scale task-specific labeled data, especially considering the inestimable number of tasks to be explored. Therefore, a rea- sonable attempt is to map diverse NLP tasks into a common learning problem—solving this com- mon problem equals to solving any downstream NLP tasks, even some tasks that are new or have insufficient annotations. Textual entailment (aka. natural language in- ference in Bowman et al. (2015)) is the task of studying the relation of two assertive sentences, Premise (P) and Hypothesis (H): whether H is true given P. Textual entailment (TE) was originally brought …

Content truncated to 500 words