On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8554–8572 July 9-14, 2023 ©2023 Association for Computational Linguistics On “Scientific Debt” in NLP: A Case for More Rigour in Language Model Pre-Training Research Made Nindyatama Nityasya1, Haryo Akbarianto Wibowo1, Alham Fikri Aji2, Genta Indra Winata3, Radityo Eko Prasojo4, Phil Blunsom5,6, Adhiguna Kuncoro7 1Independent Researcher 2MBZUAI 3Bloomberg 4Universitas Indonesia 5Cohere.AI 6University of Oxford 7DeepMind {made.nindyatama,haryo.akbarianto}@gmail.com, alham.fikri@mbzuai.ac.ae, gwinata@bloomberg.net, radityo.ep@ui.ac.id, phil.blunsom@cs.ox.ac.uk, akuncoro@deepmind.com Abstract This evidence-based position paper critiques current research practices within the language model pre-training literature. Despite rapid re- cent progress afforded by increasingly better pre-trained language models (PLMs), current PLM research practices often conflate different possible sources of model improvement, with- out conducting proper ablation studies and prin- cipled comparisons between different models under comparable conditions. These practices (i) leave us ill-equipped to understand which pre-training approaches should be used under what circumstances; (ii) impede reproducibility and credit assignment; and (iii) render it dif- ficult to understand: “How exactly does each factor contribute to the progress that we have today?” We provide a case in point by revis- iting the success of BERT over its baselines, ELMo and GPT-1, and demonstrate how — un- der comparable conditions where the baselines are tuned to a similar extent — these baselines (and even-simpler variants thereof) can, in fact, achieve competitive or better performance than BERT. These findings demonstrate how disen- tangling different factors of model improve- ments can lead to valuable new insights. We conclude with recommendations for how to en- courage and incentivize this line of work, and accelerate progress towards a better and more systematic understanding of what factors drive the progress of our foundation models today. 1 Introduction In recent years, language models that are pre- trained on large amounts of data have become the foundation models (Bommasani et al., 2021) for achieving state-of-the-art results on many NLP tasks (Peters et al., 2018; Devlin et al., 2019; Raffel et al., 2020; Liu et al., 2019, inter alia), and per- formed well at novel tasks through demonstrations (Brown et al., 2020). Hence, given the vast poten- tial of pre-trained language models (PLMs), sub- stantial effort has since been dedicated into develop- ing the next state-of-the-art PLM, whether through new objective functions (Devlin et al., 2019; Yang et al., 2019; Clark et al., 2020; Raffel et al., 2020; Tay et al., 2022, inter alia), increasing model size (Kaplan et al., 2020; Brown et al., 2020; Chowd- hery et al., 2022), better data filtering (Lee et al., 2022), or making a better use of the compute bud- get by training longer (Hoffmann et al., 2022). To a large extent, this rapid progress is made possible by a strong emphasis on designing bet- ter foundation models and achieving new state-of- the-art results. In pursuit of this goal, each new PLM often leverages all possible means for im- proving performance: Indeed, it is often the case that a new state-of-the-art PLM paper not only pro- posed a …

Content truncated to 500 words