Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2422 - 2437 December 7-11, 2022 ©2022 Association for Computational Linguistics Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrations Kang Min Yoo∗#†‡§, Junyeob Kim∗§, Hyuhng Joon Kim§, Hyunsoo Cho§, Hwiyeol Jo‡, Sang-Woo Lee†‡♮, Sang-goo Lee§, Taeuk Kim#¶ §Seoul National University, †NAVER AI Lab, ‡NAVER CLOVA ♮Korea Advanced Institute of Science and Technology, ¶Hanyang University {juny116,heyjoonkim,johyunsoo,sglee}@europa.snu.ac.kr {hwiyeol.jo,sang.woo.lee,kangmin.yoo}@navercorp.com kimtaeuk@hanyang.ac.kr Abstract Despite recent explosion of interests in in- context learning, the underlying mechanism and the precise impact of the quality of demon- strations remain elusive. Intuitively, ground- truth labels should have as much impact in in-context learning (ICL) as supervised learn- ing, but recent work reported that the input- label correspondence is significantly less impor- tant than previously thought. Intrigued by this counter-intuitive observation, we re-examine the importance of ground-truth labels in in- context learning. With the introduction of two novel metrics, namely Label-Correctness Sen- sitivity and Ground-truth Label Effect Ratio (GLER), we were able to conduct quantifiable analysis on the impact of ground-truth label demonstrations. Through extensive analyses, we find that the correct input-label mappings can have varying impacts on the downstream in-context learning performances, depending on the experimental configuration. Through additional studies, we identify key components, such as the verbosity of prompt templates and the language model size, as the controlling fac- tor to achieve more noise-resilient ICL. 1 Introduction Large-scale language models (Rae et al., 2021; Chowdhery et al., 2022; Smith et al., 2022; Thop- pilan et al., 2022) have shaped the NLP scene by introducing in-context learning (ICL) (Brown et al., 2020) as a novel approach to adapt language models for downstream tasks without explicit fine- tuning. ICL enables language models to learn and predict from task-specific prompts that con- tain demonstrations in the natural language format, *Equal contributions. #Co-corresponding authors. GPT-J SST2 GPT-J hate_speech18 GPT-3 SST2 GPT-3 TREC 0 10 20 30 40 50 60 70 80 90 100 Accuracy no demo 0% 25% 50% 75% 100% random label Figure 1: A demonstration of cases where the effect of the ground-truth label in in-context learning is much more significant than the aggregated results reported by Min et al. (2022b). despite the language models were only trained to predict the next word token. Inspired by the new discovery, a flurry of recent work has investigated ways to explain and exploit the ICL mechanism (Schick and Schütze (2021a); Lu et al. (2022); in- ter alia), but it remains elusive. Min et al. (2022b) have recently re-evaluated the role of input-label correspondence in demonstra- tions for ICL. Specifically, the authors have shown that the correct mapping between input and its label contributes less to the final performance than we thought compared to other aspects, including the format of demonstrations and the awareness of the input and label space. This finding is intriguing and has been sensational, as it is counter-intuitive to the expectation of how statistical learning typ- ically works in supervised settings, and therefore it shows …

Content truncated to 500 words