What’s the Meaning of Superhuman Performance in Today’s NLU?

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 12471–12491 July 9-14, 2023 ©2023 Association for Computational Linguistics What’s the Meaning of Superhuman Performance in Today’s NLU? Simone Tedeschi1,2, Johan Bos3, Thierry Declerck4, Jan Hajiˇc5, Daniel Hershcovich6, Eduard H. Hovy7,8, Alexander Koller9, Simon Krek10,11, Steven Schockaert12, Rico Sennrich13,14, Ekaterina Shutova15, Roberto Navigli2 1Babelscape 2Sapienza University of Rome 3University of Groningen 4German Research Center for AI (DFKI) 5Charles University 6University of Copenhagen 7University of Melbourne 8Carnegie Mellon University 9Saarland University 10Jožef Stefan Institute 11University of Ljubljana 12Cardiff University 13University of Zurich 14University of Edinburgh 15University of Amsterdam {tedeschi, navigli}@diag.uniroma1.it, johan.bos@rug.nl declerck@dfki.de hajic@ufal.mff.cuni.cz dh@di.ku.dk hovy@cmu.edu koller@coli.uni-saarland.de simon.krek@ijs.si schockaerts1@cardiff.ac.uk sennrich@cl.uzh.ch e.shutova@uva.nl Abstract In the last five years, there has been a significant focus in Natural Language Processing (NLP) on developing larger Pretrained Language Mod- els (PLMs) and introducing benchmarks such as SuperGLUE and SQuAD to measure their abilities in language understanding, reasoning, and reading comprehension. These PLMs have achieved impressive results on these bench- marks, even surpassing human performance in some cases. This has led to claims of superhu- man capabilities and the provocative idea that certain tasks have been solved. In this position paper, we take a critical look at these claims and ask whether PLMs truly have superhuman abili- ties and what the current benchmarks are really evaluating. We show that these benchmarks have serious limitations affecting the compar- ison between humans and PLMs and provide recommendations for fairer and more transpar- ent benchmarks. 1 Introduction In recent years, research in the field of Natural Lan- guage Processing (NLP) has been driven by a fran- tic race to reach the top spot in popular benchmarks (Wang et al., 2018, 2019; Lai et al., 2017; Rajpurkar et al., 2018; Reddy et al., 2019). Typically the race takes the shape of a rapid cycle of parameter tuning updates by several teams, communicating their re- sults using a shared leaderboard. Not infrequently, systems achieve better-than-human performance on several tasks (see Figure 1). Yet what does this level of performance really mean for NLP? The impressive capabilities of ChatGPT make this ques- tion even more urgent. Figure 1: Difference between the scores obtained by the best-performing systems and humans in various popular NLP benchmarks. The systems outperform humans on 6 out of 8 of the reported benchmarks (best seen in color). It is relatively easy to outperform humans with simple procedural tasks like arithmetic and extreme memory-intensive tasks involving vast amounts of data. But most tasks involving natural language typically require knowledge and inference. Do high-performing NLP algorithms really have (su- per)human capabilities? Or are the metrics that deliver these scores suspect? Given the impact of claiming superhuman perfor- mance, it is important for researchers to understand exactly what is going on. As many in NLP have experienced, the false sense of accomplishment of superhuman performance often leads to an abrupt disappointment when a supposedly superb system is applied to realistic data in a real-world situa- tion. …

Content truncated to 500 words