On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 94–102 May 2-6, 2023 ©2023 Association for Computational Linguistics WINODICT: Probing language models for in-context word acquisition Julian Martin Eisenschlos1, Jeremy R. Cole1, Fangyu Liu2, William W. Cohen1 1 Google Research 2 University of Cambridge {eisenjulian,jrcole,wcohen}@google.com fl399@cam.ac.uk Abstract We introduce a new in-context learning paradigm to measure Large Language Models’ (LLMs) ability to learn novel words during in- ference. In particular, we rewrite Winograd- style co-reference resolution problems by re- placing the key concept word with a synthetic but plausible word that the model must under- stand to complete the task. Solving this task re- quires the model to make use of the dictionary definition of the new word given in the prompt. This benchmark addresses word acquisition, one important aspect of the diachronic degra- dation known to afflict LLMs. As LLMs are frozen in time at the moment they are trained, they are normally unable to reflect the way language changes over time. We show that the accuracy of LLMs compared to the origi- nal Winograd tasks decreases radically in our benchmark, thus identifying a limitation of current models and providing a benchmark to measure future improvements in LLMs ability to do in-context learning. 1 Introduction Large Language Models (LLMs) such as GPT- 3 (Brown et al., 2020) and PALM (Chowdhery et al., 2022) can only learn from information that is in their training corpus. However, this is nat- urally limiting because the training corpus itself is bounded in time to the point of its collection. As a result, recent work has studied how to adapt such models to new data without an expensive re- training phase. Methods range from using semi- parametric methods with access to external mem- ory (e.g., Guu et al. 2020; Lewis et al. 2020), to con- tinual learning (e.g., Dhingra et al. 2022; Lazaridou et al. 2021), to parameter efficient fine-tuning (e.g., Ben Zaken et al. 2022; Pfeiffer et al. 2021). Much of this work concerns factual knowledge or task distribution shifts. However, language also changes subtly: for instance, the popularity or meaning of individual words can change over time. In fact, such shifts also cause a consistent de- crease in model performance for downstream tasks (Huang and Paul, 2018; Jaidka et al., 2018; Lukes and Søgaard, 2018; Florio et al., 2020). Acquiring new words through either examples or definitions is therefore an important test of LLMs’ ability to overcome diachronic degradation. With in-context learning having emerged as the primary way to interact with LLMs (Brown et al., 2020), we propose to study LLMs capability of acquiring new vocabulary via prompting. We propose WINODICT, a novel benchmark for word acquisition for LLMs. Word acquisition is challenging to study in a realistic setting as it is hard to know which terms a model has already been exposed to. To overcome this, we rely on a heuris- tic method to introduce newly invented words and define them in terms of …

Content truncated to 500 words