FLamE: Few-shot Learning from Natural Language Explanations

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6743–6763 July 9-14, 2023 ©2023 Association for Computational Linguistics 🔥FLamE: Few-shot Learning from Natural Language Explanations Yangqiaoyu Zhou Yiming Zhang Chenhao Tan University of Chicago {zhouy1, yimingz0, chenhao}@uchicago.edu Abstract Natural language explanations have the poten- tial to provide rich information that in principle guides model reasoning. Yet, recent work by Lampinen et al. (2022) has shown limited util- ity of natural language explanations in improv- ing classification. To effectively learn from explanations, we present FLamE, a two-stage few-shot learning framework that first gener- ates explanations using GPT-3, and then fine- tunes a smaller model (e.g., RoBERTa) with generated explanations. Our experiments on natural language inference demonstrate effec- tiveness over strong baselines, increasing accu- racy by 17.6% over GPT-3 Babbage and 5.7% over GPT-3 Davinci in e-SNLI. Despite improv- ing classification performance, human evalu- ation surprisingly reveals that the majority of generated explanations does not adequately jus- tify classification decisions. Additional analy- ses point to the important role of label-specific cues (e.g., “not know” for the neutral label) in generated explanations. 1 Introduction Collecting and learning from natural language ex- planations has received increasing attention in the NLP community (Wiegreffe and Marasović, 2021). The idea of learning from natural language expla- nations is especially appealing in few-shot learning because explanations can provide rich information about the task and guide model reasoning when there are limited supervision signals. Although large-scale language models (LLMs) have demonstrated a remarkable capability in few- shot learning (Brown et al., 2020; Rae et al., 2022; Chowdhery et al., 2022a), the effect of learning from natural language explanations remains mixed. On the one hand, Wei et al. (2022b) demonstrates impressive success with chain-of-thought prompt- ing, especially in arithmetic reasoning. On the other hand, in a systematic evaluation of the effect of ex- planations on in-context learning, Lampinen et al. (2022) discover only a marginal improvement from explanations, even when experimenting with mas- sive models (280B). It thus remains an open ques- tion how we can leverage LLMs to effectively learn from natural language explanations. We propose a two-stage framework (🔥FLamE) for Few-shot Learning from natural language Expla- nations. Fig. 1 gives a graphical overview of our approach. First, our framework leverages the abil- ity of large-scale language models (e.g., GPT-3) to generate explanations. Second, it uses explanation- aware prompt-based classification where we can fine-tune a smaller model (e.g., RoBERTa). The second step enables the model to tailor to the im- perfect explanations from GPT-3 and also opens up opportunities to interpret and probe the model given its transparent internals. We show that FLamE outperforms strong base- lines in natural language inference. Compared to GPT-3 finetuned with explanations, FLamE achieves higher accuracy than Babbage by 17.6% on e-SNLI and 6.9% on e-HANS, and also out- performs Davinci by 14.2% on e-SNLI and 14.3% on e-HANS. In addition, FLamE outperforms the strongest baselines that do not use explanations by 5.7% on e-SNLI and …

Content truncated to 500 words