Mark-Evaluate: Assessing Language Generation using Population Estimation Methods

Proceedings of the 28th International Conference on Computational Linguistics, pages 1963–1977 Barcelona, Spain (Online), December 8-13, 2020 1963 Mark-Evaluate: Assessing Language Generation using Population Estimation Methods Gonc¸alo Mordido Hasso Plattner Institute Potsdam, Germany goncalo.mordido@hpi.de Christoph Meinel Hasso Plattner Institute Potsdam, Germany christoph.meinel@hpi.de Abstract We propose a family of metrics to assess language generation derived from population estima- tion methods widely used in ecology. More specifically, we use mark-recapture and maximum- likelihood methods that have been applied over the past several decades to estimate the size of closed populations in the wild. We propose three novel metrics: MEPetersen and MECAPTURE, which retrieve a single-valued assessment, and MESchnabel which returns a double-valued metric to assess the evaluation set in terms of quality and diversity, separately. In synthetic experiments, our family of methods is sensitive to drops in quality and diversity. Moreover, our methods show a higher correlation to human evaluation than existing metrics on several challenging tasks, namely unconditional language generation, machine translation, and text summarization. 1 Introduction Population estimation methods have been widely used in ecology to study the development of species over the last several decades (Krebs and others, 1989). Existing population estimation methods focus on open populations, where births, deaths, and migrations are taken into account, or closed population methods, where the population is assumed to remain static over the population estimation study. In this work, we focus on closed population methods and study how their respective population estimates can be used to evaluate an evaluation set of generated samples, given a reference set of real samples. In this work, samples are either contextualized word or sentence embeddings. We study two mark- recapture methods, namely the Petersen (Ricker, 1975) and the Schnabel (Schnabel, 1938) estimators, where samples are captured and marked, released, and recaptured. The number of such different samples is then used to estimate the population size. We further use CAPTURE (Otis et al., 1978), a maximum- likelihood method, which uses the number of marked samples over multiple captures for such estimation. Accurate evaluation of generated data is essential to correctly measure in what degrees we can im- prove the overall generation process. Depending on the use case, single-valued metrics may suffice to assess specific conditional language generation tasks, such as machine translation and text summariza- tion, where we are interested in evaluating the similarities of a generated translation or summary to a specific reference translation or summary, respectively. On the other hand, on unconditional language generation, for example, it may be useful to have separate measures for the diversity and quality of the generated set, enabling the identification of possible shortcomings of our generation system and try to fix it accordingly. This has been an active area of generative models research (Goodfellow et al., 2014), with several works focusing on stimulating diversity while maintaining the overall sample quality (Srivastava et al., 2017; Lin et al., 2018; Mordido et al., 2018; Mordido et al., 2020; Sauder et al., 2020). Mark-Evaluate (ME) is a family of 3 novel language …

Content truncated to 500 words