ReasonBERT: Pre-trained to Reason with Distant Supervision

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6112–6127 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 6112 ReasonBERT: Pre-trained to Reason with Distant Supervision Xiang Deng*1, Yu Su1, Alyssa Lees2, You Wu2, Cong Yu2, and Huan Sun*1 1The Ohio State University, Columbus, OH {deng.595,su.809,sun.397}@osu.edu 2Google Research, New York, NY {alyssalees,wuyou,congyu}@google.com Abstract We present ReasonBERT, a pre-training method that augments language models with the ability to reason over long-range relations and multiple, possibly hybrid, contexts. Un- like existing pre-training methods that only harvest learning signals from local contexts of naturally occurring texts, we propose a gen- eralized notion of distant supervision to auto- matically connect multiple pieces of text and tables to create pre-training examples that re- quire long-range reasoning. Different types of reasoning are simulated, including inter- secting multiple pieces of evidence, bridging from one piece of evidence to another, and detecting unanswerable cases. We conduct a comprehensive evaluation on a variety of extractive question answering datasets rang- ing from single-hop to multi-hop and from text-only to table-only to hybrid that require various reasoning capabilities and show that ReasonBERT achieves remarkable improve- ment over an array of strong baselines. Few- shot experiments further demonstrate that our pre-training method substantially improves sample efficiency.1 1 Introduction Recent advances in pre-trained language mod- els (LMs) have remarkably transformed the land- scape of natural language processing. Pre-trained with self-supervised objectives such as autoregres- sive language modeling (Radford and Narasimhan, 2018; Radford et al., 2019; Brown et al., 2020) and masked language modeling (MLM) (Devlin et al., 2019; Liu et al., 2019b; Joshi et al., 2020), LMs encode a great deal of knowledge about language and significantly boost model performance on a wide range of downstream tasks (Liu et al., 2019a; Wang et al., 2019a,b) ranging from spell checking *Corresponding authors. 1Our code and pre-trained models are available at https: //github.com/sunlab-osu/ReasonBERT. (Awasthi et al., 2019) to sentiment analysis (Xu et al., 2019) and semantic parsing (Rongali et al., 2020), just to name a few. Existing self-supervised objectives for LM pre- training primarily focus on consecutive, naturally occurring text. For example, MLM enables LMs to correctly predict the missing word “daugh- ters” in the sentence “Obama has two __ , Malia and Sasha.” based on the local context and the knowledge stored in the parameters. However, many tasks require reasoning beyond local con- texts: multi-hop question answering (QA) (Yang et al., 2018; Welbl et al., 2018) and fact verification (Jiang et al., 2020) require reasoning over multi- ple pieces of evidence, hybrid QA (Chen et al., 2020) requires simultaneously reasoning over un- structured text and structured tables, and dialogue systems require reasoning over the whole dialogue history to accurately understand the current user utterance (Andreas et al., 2020). To address this limitation in existing LM pre- training, we propose ReasonBERT, a pre-training method to augment LMs for explicitly reasoning over long-range relations and multiple contexts. ReasonBERT pairs a query sentence with multiple relevant pieces of evidence drawn from possibly different …

Content truncated to 500 words