Few-Shot Question Answering by Pretraining Span Selection

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3066–3079 August 1–6, 2021. ©2021 Association for Computational Linguistics 3066 Few-Shot Question Answering by Pretraining Span Selection Ori Ram∗,1 Yuval Kirstain∗,1 Jonathan Berant1,2 Amir Globerson1 Omer Levy1 Blavatnik School of Computer Science, Tel Aviv University1 Allen Institute for AI2 {ori.ram,yuval.kirstain,joberant,gamir,levyomer}@cs.tau.ac.il Abstract In several question answering benchmarks, pretrained models have reached human parity through fine-tuning on an order of 100,000 an- notated questions and answers. We explore the more realistic few-shot setting, where only a few hundred training examples are avail- able, and observe that standard models per- form poorly, highlighting the discrepancy be- tween current pretraining objectives and ques- tion answering. We propose a new pretrain- ing scheme tailored for question answering: re- curring span selection. Given a passage with multiple sets of recurring spans, we mask in each set all recurring spans but one, and ask the model to select the correct span in the pas- sage for each masked span. Masked spans are replaced with a special token, viewed as a question representation, that is later used dur- ing fine-tuning to select the answer span. The resulting model obtains surprisingly good re- sults on multiple benchmarks (e.g., 72.7 F1 on SQuAD with only 128 training examples), while maintaining competitive performance in the high-resource setting.1 1 Introduction The standard approach to question answering is to pretrain a masked language model on raw text, and then fine-tune it with a span selection layer on top (Devlin et al., 2019; Joshi et al., 2020; Liu et al., 2019). While this approach is effective, and some- times exceeds human performance, its success is based on the assumption that large quantities of an- notated question answering examples are available. For instance, both SQuAD (Rajpurkar et al., 2016, 2018) and Natural Questions (Kwiatkowski et al., 2019) contain an order of 100,000 question and ∗ Equal contribution. 1Our code, models, and datasets are publicly available: https://github.com/oriram/splinter. # Examples F1 0.0 20.0 40.0 60.0 80.0 100.0 16 32 64 128 256 SpanBERT (Full Data) RoBERTa SpanBERT Splinter (Ours) Figure 1: Performance of SpanBERT (red) and RoBERTa (yellow) base-size models on SQuAD, given different amounts of training examples. Our model (Splinter, green) dramatically improves performance. SpanBERT-base trained on the full training data of SQuAD (blue, dashed) is shown for reference. answer pairs in their training data. This assump- tion quickly becomes unrealistic as we venture out- side the lab conditions of English Wikipedia, and attempt to crowdsource question-answer pairs in other languages or domains of expertise (Tsatsaro- nis et al., 2015; Kembhavi et al., 2017). How do question answering models fare in the more practi- cal case, where an in-house annotation effort can only produce a couple hundred training examples? We investigate the task of few-shot question an- swering by sampling small training sets from exist- ing question answering benchmarks. Despite the use of pretrained models, the standard approach yields poor results when fine-tuning …

Content truncated to 500 words