Prompting Language Models for Linguistic Structure

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6649–6663 July 9-14, 2023 ©2023 Association for Computational Linguistics Prompting Language Models for Linguistic Structure Terra Blevins Hila Gonen Luke Zettlemoyer Paul G. Allen School of Computer Science & Engineering, University of Washington {blvns, lsz}@cs.washington.edu hilagnn@gmail.com Abstract Although pretrained language models (PLMs) can be prompted to perform a wide range of lan- guage tasks, it remains an open question how much this ability comes from generalizable lin- guistic understanding versus surface-level lex- ical patterns. To test this, we present a struc- tured prompting approach for linguistic struc- tured prediction tasks, allowing us to perform zero- and few-shot sequence tagging with au- toregressive PLMs. We evaluate this approach on part-of-speech tagging, named entity recog- nition, and sentence chunking, demonstrating strong few-shot performance in all cases. We also find that while PLMs contain significant prior knowledge of task labels due to task leakage into the pretraining corpus, structured prompting can also retrieve linguistic structure with arbitrary labels. These findings indicate that the in-context learning ability and linguis- tic knowledge of PLMs generalizes beyond memorization of their training data. 1 Introduction The rapid increase in the scale of pretrained lan- guage models (PLMs) has led to a new paradigm of NLP modeling: in-context learning, or prompting (e.g., Brown et al., 2020; Raffel et al., 2020). In this setting, the model is used to perform a task directly via the predictions of the LM head with- out additional finetuning on the target task, often with a few demonstrations of the desired behavior provided within the input. This setup has led to impressive few-shot performance on various tasks ranging from classification to summarization and generation (Liu et al., 2021a). Due to their broad success on tasks requiring language understanding, we hypothesize that these models also contain significant linguistic knowl- edge. However, we are not aware of existing prompting methods that can directly test this hy- pothesis on autoregressive PLMs. Behavioral anal- ysis of PLMs (Belinkov et al., 2020) uses meth- C: Alright what is this ? T: Alright_INTJ what_PRON is_AUX this_PRON ?_PUNCT C: I love this color . T: I_PRON love_VERB this_DET color_NOUN ._PUNCT demonstration C: It took a while . T: It_PRON took_VERB a_ PLM example NOUN DET VERB … Figure 1: Sequence tagging via structured prompting. Each predicted label is appended to the context along with the next word to iteratively tag the full sentence. ods similar to prompting to measure knowledge stored in language models (Gulordava et al., 2018; Petroni et al., 2019), but this technique is difficult to generalize to tasks that predict more complex structures. Additionally, current approaches for applying PLMs to linguistic structured prediction tasks finetune on the downstream task (e.g., Ma et al., 2022), which confounds measuring underly- ing model knowledge. We propose a new approach, structured prompt- ing, that iteratively prompts autoregressive PLMs to probe for word- and span-level linguistics framed as sequence tagging tasks (Section 2). At timestep t, …

Content truncated to 500 words