How Can We Accelerate Progress Towards Human-like Linguistic Generalization?

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5210–5217 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5210 How Can We Accelerate Progress Towards Human-like Linguistic Generalization? Tal Linzen Department of Cognitive Science Johns Hopkins University tal.linzen@jhu.edu Abstract This position paper describes and critiques the Pretraining-Agnostic Identically Distributed (PAID) evaluation paradigm, which has be- come a central tool for measuring progress in natural language understanding. This paradigm consists of three stages: (1) pre- training of a word prediction model on a cor- pus of arbitrary size; (2) fine-tuning (transfer learning) on a training set representing a classi- fication task; (3) evaluation on a test set drawn from the same distribution as that training set. This paradigm favors simple, low-bias archi- tectures, which, first, can be scaled to process vast amounts of data, and second, can capture the fine-grained statistical properties of a par- ticular data set, regardless of whether those properties are likely to generalize to examples of the task outside the data set. This contrasts with humans, who learn language from several orders of magnitude less data than the systems favored by this evaluation paradigm, and gen- eralize to new tasks in a consistent way. We advocate for supplementing or replacing PAID with paradigms that reward architectures that generalize as quickly and robustly as humans. 1 Introduction The special session of the 2020 Annual Meeting of Association for Computational Linguistics in- vites us to take stock of the progress made in the field in the last few years. There is no question that we have made significant progress in a range of applications: current machine translation sys- tems for high-resource languages, for example, are undeniably better than those we had a decade ago. This opinion piece will focus on a different ques- tion: are we making progress towards the classic goal of mimicking human linguistic abilities in machines—towards a model that acquires language as efficiently as humans, and generalizes it as hu- mans do to new structures and contexts (“tasks”)? I will argue that an evaluation paradigm that has rapidly established itself as one of the main tools for measuring progress in the field—a paradigm I will term, for want of a catchier name, Pretraining-Agnostic Identically Distributed evalu- ation (PAID)—encourages progress in a direction that is at best orthogonal to the goal of human-like generalization. Because it does not consider sam- ple efficiency, this approach rewards models that can be trained on massive amounts of data, sev- eral orders of magnitude more than a human can expect to be exposed to. And because benchmark scores are computed on test sets drawn from the same distribution as their respective training sets, this paradigm favors models that excel in capturing the statistical patterns of particular data sets over models that generalize as a human would. 2 Human-like Generalization Humans learn language from much more limited exposure than most contemporary NLP systems. An analysis of recordings taken in the environment of the child of an …

Content truncated to 500 words