Retrieval-guided Counterfactual Generation for QA

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1670 - 1686 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Retrieval-guided Counterfactual Generation for QA Bhargavi Paranjape1∗, Matthew Lamm2, and Ian Tenney2 1Paul G. Allen School of Computer Science & Engineering, University of Washington 2Google Research bparan@cs.washington.edu {mrlamm,iftenney}@google.com Abstract Deep NLP models have been shown to be brit- tle to input perturbations. Recent work has shown that data augmentation using counter- factuals — i.e. minimally perturbed inputs — can help ameliorate this weakness. We focus on the task of creating counterfactuals for ques- tion answering, which presents unique chal- lenges related to world knowledge, semantic diversity, and answerability. To address these challenges, we develop a Retrieve-Generate- Filter (RGF) technique to create counterfac- tual evaluation and training data with minimal human supervision. Using an open-domain QA framework and question generation model trained on original task data, we create coun- terfactuals that are fluent, semantically diverse, and automatically labeled. Data augmenta- tion with RGF counterfactuals improves per- formance on out-of-domain and challenging evaluation sets over and above existing meth- ods, in both the reading comprehension and open-domain QA settings. Moreover, we find that RGF data leads to significant improve- ments to robustness to local perturbations.1 1 Introduction Models for natural language understanding (NLU) may outperform humans on standard benchmarks, yet still often perform poorly under a multitude of distributional shifts (Jia and Liang (2017); Naik et al. (2018); McCoy et al. (2019), inter alia) due to over-reliance on spurious correlations or dataset artifacts. This behavior can be probed using coun- terfactual data (Kaushik et al., 2020; Gardner et al., 2020) designed to simulate interventions on spe- cific attributes: for example, perturbing the movie review “A real stinker, one out of ten!" to “A real classic, ten out of ten!" allows us to discern the ∗Work performed during an internship at Google. 1Code at https://github.com/ google-research/language/tree/master/ language/qa_counterfactuals RETRIEVE (REALM) GENERATE FILTER Wikipedia Who is the captain of the Richmond Football Club? List of Richmond Football Club captains >> Jeff Hogg 1994 -- 1996 ... Current Captain: Trent Cotchin The ... Richmond Football club ran a women's team and ... Jess Kennedy was named the team ' s captain Who captained Richmond Football Club's women's team? Who won the inaugural best in VFL 2018 season? Who captained Richmond Football Club's women's team? Figure 1: Retrieve-Generate-Filter to generate coun- terfactual queries for Natural Question (Kwiatkowski et al., 2019) using an open-domain retrieval system, question generation and post-hoc filtering. effect of adjective polarity on the model’s predic- tion. Many recent works (Kaushik et al., 2020, 2021; Wu et al., 2021a; Geva et al., 2021, inter alia) have shown that training augmented with this counterfactual data (CDA) improves out-of-domain generalization and robustness against spurious cor- relations. Consequently, several techniques have been proposed for the automatic generation of coun- terfactual data for several downstream tasks (Wu et al., 2021a; Ross et al., 2021b,a; Bitton et al., 2021; Geva et al., …

Content truncated to 500 words