Reframing Human-AI Collaboration for Generating Free-Text Explanations

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 632 - 658 July 10-15, 2022 ©2022 Association for Computational Linguistics Reframing Human-AI Collaboration for Generating Free-Text Explanations Sarah Wiegreffe♣ Jack Hessel† Swabha Swayamdipta† Mark Riedl♣ Yejin Choi3† ♣School of Interactive Computing, Georgia Institute of Technology †Allen Institute for Artificial Intelligence 3Paul G. Allen School of Computer Science and Engineering, University of Washington saw@gatech.edu, riedl@cc.gatech.edu, {jackh,swabhas,yejinc}@allenai.org Abstract Large language models are increasingly capa- ble of generating fluent-appearing text with rel- atively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using human-written ex- amples in a few-shot manner. We find that (1) authoring higher quality prompts results in higher quality generations; and (2) surpris- ingly, in a head-to-head comparison, crowd- workers often prefer explanations generated by GPT-3 to crowdsourced explanations in exist- ing datasets. Our human studies also show, however, that while models often produce fac- tual, grammatical, and sufficient explanations, they have room to improve along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorpo- rates binary acceptability judgments from hu- mans in the loop. Despite the intrinsic subjec- tivity of acceptability judgments, we demon- strate that acceptability is partially correlated with various fine-grained attributes of explana- tions. Our approach is able to consistently fil- ter GPT-3-generated explanations deemed ac- ceptable by humans. 1 Introduction As natural language understanding tasks have be- come increasingly complex, the field of explainable NLP has embraced explanations written in free- form natural language. In contrast to extractive explanations that highlight tokens in the input, free- text explanations provide a natural interface be- tween machine computation and human end-users (Hendricks et al., 2016; Camburu et al., 2018). The dominant paradigm for producing free-text expla- nations is via direct supervision, i.e., training an autoregressive, generative language model to pre- dict human-authored explanations directly (Kim et al., 2018; Park et al., 2018; Ehsan et al., 2018; Narang et al., 2020; Wiegreffe et al., 2021, i.a.). GPT-3 DaVinci T5 / T0 Filter Model Overgenerating Explanation Candidates top score Select-1 Explanation-Level Filtering Acceptable Explanations Evaluation Instance + Explanation Prompt Acceptability Predictions greedy stochastic Crowdworker Acceptability Labels “When eating a hamburger with friends, what is one trying to do?” have fun. Explanation: … “Usually a hamburger with friends indicates a good time.” Figure 1: Illustration of our overgeneration + filtra- tion pipeline for producing human acceptable explana- tions for CommonsenseQA and SNLI (see examples in Table 1). Authors of this work write explanations to prompt GPT-3, generating 5 explanations per instance. An acceptability filter, trained on human binary accept- ability judgments, determines which of these generated explanations are plausible. Evaluation is performed at both the explanation and the instance level. However, collecting high-quality written expla- nations to serve as supervision is difficult and ex- pensive. More than 70% of existing free-text ex- planation datasets are …

Content truncated to 500 words