Benchmarking Large Language Model Capabilities for Conditional Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9194–9213 July 9-14, 2023 ©2023 Association for Computational Linguistics Benchmarking Large Language Model Capabilities for Conditional Generation Joshua Maynez Google DeepMind joshuahm@google.com Priyanka Agrawal Google DeepMind priyankagr@google.com Sebastian Gehrmann Google Research gehrmann@google.com Abstract Pre-trained large language models (PLMs) un- derlie most new developments in natural lan- guage processing. They have shifted the field from application-specific model pipelines to a single model that is adapted to a wide range of tasks. Autoregressive PLMs like GPT-3 or PaLM, alongside techniques like few-shot learning, have additionally shifted the output modality to generation instead of classifica- tion or regression. Despite their ubiquitous use, the generation quality of language mod- els is rarely evaluated when these models are introduced. Additionally, it is unclear how ex- isting generation tasks—-while they can be used to compare systems at a high level—- relate to the real world use cases for which people have been adopting them. In this work, we discuss how to adapt existing application- specific generation benchmarks to PLMs and provide an in-depth, empirical study of the limitations and capabilities of PLMs in nat- ural language generation tasks along dimen- sions such as scale, architecture, input and out- put language. Our results show that PLMs differ in their applicability to different data regimes and their generalization to multiple languages and inform which PLMs to use for a given generation task setup. We share best practices to be taken into consideration when benchmarking generation capabilities during the development of upcoming PLMs. 1 Introduction Natural language generation tasks require gener- ating understandable text given textual or non- linguistic information as input, such as documents, tables, or other structured forms. These texts seek to achieve a communicative goal (e.g., summarize a document). The standard approach to tackle these problems over the last years has been to start with a pretrained encoder-decoder model like T5 (Raffel et al., 2020a) or BART (Lewis et al., 2020a) and finetune it on a corpus that captures the downstream task. The recent much larger pretrained language models use a decoder-only architecture and upended this paradigm. These models enabled few-shot or in-context learning ap- proaches in which a model is presented with one or more examples and tasked to continue generating without any finetuning. We refer to both kinds of pretrained models as PLMs. Due to the lack of grounding in the specific task setup, few-shot learning in generation settings leads to a model approaching the communicative goal from very different angles. These diverse range of outputs make the typical reference-based automatic evaluation strategies largely incompati- ble. While human evaluation can be used to over- come this shortcoming, it is infeasible to monitor the performance of an actively training model this way or to re-run all evaluations every time a new model is introduced. This leads to the question how one should reliably monitor generation ca- pabilities, a question that is only growing in im- portance as more …

Content truncated to 500 words