Language Model Evaluation Beyond Perplexity

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 5328–5339 August 1–6, 2021. ©2021 Association for Computational Linguistics 5328 Language Model Evaluation Beyond Perplexity Clara Meister Ryan Cotterell , ETH Z¨urich University of Cambridge {first.last}@inf.ethz.ch Abstract We propose an alternate approach to quanti- fying how well language models learn natu- ral language: we ask how well they match the statistical tendencies of natural language. To answer this question, we analyze whether text generated from language models exhibits the statistical tendencies present in the human- generated text on which they were trained. We provide a framework—paired with signif- icance tests—for evaluating the fit of language models to these trends. We find that neural language models appear to learn only a subset of the tendencies considered, but align much more closely with empirical trends than pro- posed theoretical distributions (when present). Further, the fit to different distributions is highly-dependent on both model architecture and generation strategy. As concrete exam- ples, text generated under the nucleus sam- pling scheme adheres more closely to the type– token relationship of natural language than text produced using standard ancestral sam- pling; text from LSTMs reflects the natural language distributions over length, stopwords, and symbols surprisingly well. 1 Introduction Neural language models1 have become shockingly good at modeling natural language data in recent years (Merity et al., 2017; Conneau and Lample, 2019; Radford et al., 2019). Thus, to test just how well neural language models capture language NLP researchers have started to look beyond standard evaluation metrics such as perplexity, endeavoring to understand which underlying attributes of human language these models are learning. To this end, a nascent literature has emerged that focuses on probing language models (Belinkov 1In this work, we do not use the term language model to refer to cloze language models such as BERT (Devlin et al., 2019), which do not give us a distribution over strings. Figure 1: Average number of unique words vs. document length, i.e., type–token, in text sampled from language models. Values from models’ test set are plotted for reference. and Glass, 2019), i.e., determining whether models encode linguistic phenomena. For the most part, these works have been limited to analyses of sentence-level phenomenon, such as subject–verb agreement (Gulordava et al., 2018) and garden path effects (van Schijndel and Linzen, 2018) among a myriad of other properties (Blevins et al., 2018; Chowdhury and Zamparelli, 2018, inter alia). In this work, we attempt to understand which macro-level phenomena of human language today’s language models reflect. That is, we pose the ques- tion: Do neural language models exhibit the sta- tistical tendencies of human language? Phenom- ena that can be measured at this level provide an alternate view of a model’s comprehension; for example, rather than exploring whether morpho- logical agreement is captured, we look at whether our models learn the trends across a corpus as a whole, e.g., the token rank–frequency (Zipf’s) re- lationship. In comparison …

Content truncated to 500 words