Span Selection Pre-training for Question Answering

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2773–2782 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2773 Span Selection Pre-training for Question Answering Michael Glass,1 Alfio Gliozzo,1 Rishav Chakravarti,1 Anthony Ferritto,1 Lin Pan,1 G P Shrivatsa Bhargav,2 Dinesh Garg,1 Avirup Sil1 1 IBM Research AI 2 Dept. of CSA, IISC, Bangalore mrglass@us.ibm.com, gliozzo@us.ibm.com, rchakravarti@us.ibm.com, aferritto@ibm.com, panl@us.ibm.com, bhargavs@iisc.ac.in, garg.dinesh@in.ibm.com, avi@us.ibm.com Abstract BERT (Bidirectional Encoder Representations from Transformers) and related pre-trained Transformers have provided large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). BERT is pre- trained on two auxiliary tasks: Masked Lan- guage Model and Next Sentence Prediction. In this paper we introduce a new pre-training task inspired by reading comprehension to better align the pre-training from memoriza- tion to understanding. Span Selection Pre- Training (SSPT) poses cloze-like training in- stances, but rather than draw the answer from the model’s parameters, it is selected from a relevant passage. We find significant and con- sistent improvements over both BERTBASE and BERTLARGE on multiple Machine Reading Comprehension (MRC) datasets. Specifically, our proposed model has strong empirical ev- idence as it obtains SOTA results on Natu- ral Questions, a new benchmark MRC dataset, outperforming BERTLARGE by 3 F1 points on short answer prediction. We also show signif- icant impact in HotpotQA, improving answer prediction F1 by 4 points and supporting fact prediction F1 by 1 point and outperforming the previous best system. Moreover, we show that our pre-training approach is particularly effec- tive when training data is limited, improving the learning curve by a large amount. 1 Introduction State-of-the-art approaches for NLP tasks are based on language models that are pre-trained on tasks which do not require labeled data (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Sun et al., 2019). Fine tuning language models to downstream tasks, such as question answering or other natural lan- guage understanding tasks, has been shown to be a general and effective strategy. BERT is a recently introduced and highly successful model for lan- guage understanding. The general BERT adaptation approach is to alter the model used for pre-training while retaining the transformer encoder layers. The model discards the layers used for the final prediction in the pre- training tasks and adds layers to predict the target task. All parameters are then fine tuned on the target task. BERT is based on the transformer architec- ture (Vaswani et al., 2017), and trained on the fol- lowing two unsupervised tasks: • Masked Language Model (MLM): predicting masked word pieces from the surrounding con- text • Next Sentence Prediction (NSP): predicting if the two provided sequences follow sequen- tially in text or not The masked LM or “cloze” task (Taylor, 1953) and next sentence prediction are auxiliary tasks (Ando and Zhang, 2005) requiring language understanding, and therefore train the model to acquire effective representations of language. How- ever, the cloze pre-training task often poses in- stances that …

Content truncated to 500 words