Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3968–3981, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 3968 Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation Cristina Gˆarbacea1, Samuel Carton2, Shiyan Yan2, Qiaozhu Mei1,2 1Department of EECS, University of Michigan, Ann Arbor, MI, USA 2School of Information, University of Michigan, Ann Arbor, MI, USA {garbacea, scarton, shiyansi, qmei}@umich.edu Abstract We conduct a large-scale, systematic study to evaluate the existing evaluation methods for natural language generation in the con- text of generating online product reviews. We compare human-based evaluators with a va- riety of automated evaluation procedures, in- cluding discriminative evaluators that measure how well machine-generated text can be dis- tinguished from human-written text, as well as word overlap metrics that assess how similar the generated text compares to human-written references. We determine to what extent these different evaluators agree on the ranking of a dozen of state-of-the-art generators for online product reviews. We find that human eval- uators do not correlate well with discrimina- tive evaluators, leaving a bigger question of whether adversarial accuracy is the correct ob- jective for natural language generation. In gen- eral, distinguishing machine-generated text is challenging even for human evaluators, and human decisions correlate better with lexical overlaps. We find lexical diversity an intrigu- ing metric that is indicative of the assessments of different evaluators. A post-experiment sur- vey of participants provides insights into how to evaluate and improve the quality of natural language generation systems 1. 1 Introduction Recent developments in neural language models (Mikolov and Zweig, 2012), (Reiter and Belz, 2009), (Mikolov et al., 2011b), (Mikolov et al., 2011a) have inspired the use of neural network based architectures for the task of natural language generation (NLG). Despite fast development of al- gorithms, there is an urgency to fill the huge gap in evaluating NLG systems. On one hand, a rigorous, 1 The experimental setup, data, and annotations are pub- licly available at: https://github.com/Crista23/ JudgeTheJudges efficient, and reproducible evaluation procedure is critical for the development of any machine learn- ing technology and for correct interpretation of the state-of-the-art. On the other hand, evaluating the quality of language generation is inherently dif- ficult due to the special properties of text, such as subjectivity and non-compositionality. Indeed, “there is no agreed objective criterion for comparing the goodness of texts” (Dale and Mellish, 1998), and there lacks a clear model of text quality (Hard- castle and Scott, 2008). Conventionally, most NLG systems have been evaluated in a rather informal manner. (Reiter and Belz, 2009) divide existing evaluation meth- ods commonly employed in text generation into three categories: i) evaluations based on task per- formance, ii) human judgments and ratings, where human subjects are recruited to rate different di- mensions of the generated texts, and iii) evalua- tions based on comparison to a reference corpus using automated metrics. Task based evaluation considers that …

Content truncated to 500 words