FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6081–6090 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6081 FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models Rakesh Chada Amazon Alexa AI rakchada@amazon.com Pradeep Natarajan Amazon Alexa AI natarap@amazon.com Abstract The task of learning from only a few exam- ples (called a few-shot setting) is of key im- portance and relevance to a real-world set- ting. For question answering (QA), the cur- rent state-of-the-art pre-trained models typi- cally need fine-tuning on tens of thousands of examples to obtain good results. Their perfor- mance degrades significantly in a few-shot set- ting (< 100 examples). To address this, we propose a simple fine-tuning framework that leverages pre-trained text-to-text models and is directly aligned with their pre-training frame- work. Specifically, we construct the input as a concatenation of the question, a mask token representing the answer span and a context. Given this input, the model is fine-tuned using the same objective as that of its pre-training objective. Through experimental studies on various few-shot configurations, we show that this formulation leads to significant gains on multiple QA benchmarks (an absolute gain of 34.2 F1 points on average when there are only 16 training examples). The gains extend fur- ther when used with larger models (Eg:- 72.3 F1 on SQuAD using BART-large with only 32 examples) and translate well to a multilin- gual setting . On the multilingual TydiQA benchmark, our model outperforms the XLM- Roberta-large by an absolute margin of upto 40 F1 points and an average of 33 F1 points in a few-shot setting (<= 64 training examples). We conduct detailed ablation studies to ana- lyze factors contributing to these gains. 1 Introduction The task of question answering (QA) in Natural Language Processing typically involves producing an answer for a given question using a context that contains evidence to support the answer. The latest advances in pre-trained language models resulted in performance close to (and sometimes exceeding) a human performance when fine-tuned on several QA benchmarks (Devlin et al., 2019), (Brown et al., 2020), (Bao et al., 2020), (Raffel et al., 2020). How- ever, to achieve this result, these models need to be fine-tuned on tens of thousands of examples. In a more realistic and practical scenario, where only a handful of annotated training examples are avail- able, their performance degrades significantly. For instance, (Ram et al., 2021) show that, when only 16 training examples are available, the Roberta- base (Liu et al., 2019) and SpanBERT-base (Joshi et al., 2020) obtain a F1 score of 7.7 and 18.2 respectively on SQuAD (Rajpurkar et al., 2016). This is far lower than the F1 score of 90.3 and 92.0 when using the full training set of >100000 examples. Through experimental analysis, we ob- serve that this degradation is majorly attributed to the disparities between fine-tuning and pre-training frameworks (a combination of the input-output de- sign and the training objective). And to address this, we …

Content truncated to 500 words