Unifying Human and Statistical Evaluation for Natural Language Generation

Proceedings of NAACL-HLT 2019, pages 1689–1701 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 1689 Unifying Human and Statistical Evaluation for Natural Language Generation Tatsunori B. Hashimoto*1,2 Hugh Zhang*1 Percy Liang1,2 (* equal contribution) 1Department of Computer Science 2Department of Statistics Stanford University {thashim,hughz}@stanford.edu pliang@cs.stanford.edu Abstract How can we measure whether a natural lan- guage generation system produces both high quality and diverse outputs? Human evalu- ation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, sta- tistical evaluation (i.e., perplexity) captures di- versity but not quality, as models that occa- sionally emit low quality samples would be in- sufficiently penalized. In this paper, we pro- pose a unified framework which evaluates both diversity and quality, based on the optimal er- ror rate of predicting whether a sentence is human- or machine-generated. We demon- strate that this error rate can be efficiently es- timated by combining human and statistical evaluation, using an evaluation metric which we call HUSE. On summarization and chit- chat dialogue, we show that (i) HUSE detects diversity defects which fool pure human evalu- ation and that (ii) techniques such as annealing for improving quality actually decrease HUSE due to decreased diversity. 1 Introduction Generating text is a core part of many NLP tasks such as image captioning (Lin et al., 2014), open- domain dialogue (Sordoni et al., 2015), story gen- eration (Roemmele, 2016), and summarization (Nallapati et al., 2016). However, proper evalu- ation of natural language generation has proven difficult (Liu et al., 2016; Novikova et al., 2017; Chaganty et al., 2018). A good evaluation metric should not only capture the quality of generation, but also the diversity of generation, which is es- pecially crucial for creative, open-ended tasks like dialogue or story generation. Human evaluation, which is often viewed as the gold standard evaluation, captures quality but fails to capture diversity. As an example, for language Reference Model Probability (pmodel) Agassi bows out of Australian open Agassi withdraws from Australian open Sharon has stroke for stroke Cleared coach facing another grilling from British swim bosses Model Generations Reference Human Judgment Figure 1: HUSE is twice the classification error of dis- tinguishing reference and generated text based on hu- man judgment scores and model probabilities. HUSE identifies samples with defects in quality (Sharon has stroke . . .) and diversity (Cleared coach facing . . .). modeling, a model that directly plagiarizes sen- tences from the training set would pass the hu- man quality bar but would have zero generaliza- tion ability and thus have inadequate diversity. On the other hand, statistical evaluation—i.e., per- plexity on a reference test set—captures diversity, as it ensures a model must assign reasonable prob- ability to novel sentences, but perplexity provides an inadequate measure of quality (Theis et al., 2015). For example, modifying a perfect model by removing its ability to generate even a single test sentence results in infinite perplexity even …

Content truncated to 500 words