MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 563–578, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 563 MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance Wei Zhao†, Maxime Peyrard†, Fei Liu‡, Yang Gao†, Christian M. Meyer†, Steffen Eger† † Computer Science Department, Technische Universit¨at Darmstadt, Germany ‡ Computer Science Department, University of Central Florida, US zhao@aiphes.tu-darmstadt.de, maxime.peyrard@epfl.ch feiliu@cs.ucf.edu, yang.gao@rhul.ac.uk meyer@ukp.informatik.tu-darmstadt.de eger@aiphes.tu-darmstadt.de Abstract A robust evaluation metric has a profound im- pact on the development of text generation sys- tems. A desirable metric compares system out- put against references based on their seman- tics rather than surface forms. In this paper we investigate strategies to encode system and reference texts to devise a metric that shows a high correlation with human judgment of text quality. We validate our new metric, namely MoverScore, on a number of text generation tasks including summarization, machine trans- lation, image captioning, and data-to-text gen- eration, where the outputs are produced by a variety of neural and non-neural systems. Our findings suggest that metrics combining contextualized representations with a distance measure perform the best. Such metrics also demonstrate strong generalization capability across tasks. For ease-of-use we make our metrics available as web service.1 1 Introduction The choice of evaluation metric has a significant impact on the assessed quality of natural language outputs generated by a system. A desirable met- ric assigns a single, real-valued score to the sys- tem output by comparing it with one or more ref- erence texts for content matching. Many natural language generation (NLG) tasks can benefit from robust and unbiased evaluation, including text- to-text (machine translation and summarization), data-to-text (response generation), and image-to- text (captioning) (Gatt and Krahmer, 2018). With- out proper evaluation, it can be difficult to judge on system competitiveness, hindering the develop- ment of advanced algorithms for text generation. It is an increasingly pressing priority to develop better evaluation metrics given the recent advances in neural text generation. Neural models provide 1Our code is publicly available at http://tiny.cc/vsqtbz the flexibility to copy content from source text as well as generating unseen words (See et al., 2017). This aspect is hardly covered by existing metrics. With greater flexibility comes increased demand for unbiased evaluation. Diversity-promoting ob- jectives make it possible to generate diverse nat- ural language descriptions (Li et al., 2016; Wise- man et al., 2018). But standard evaluation met- rics including BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) compute the scores based pri- marily on n-gram co-occurrence statistics, which are originally proposed for diagnostic evaluation of systems but not capable of evaluating text qual- ity (Reiter, 2018), as they are not designed to mea- sure if, and to what extent, the system and refer- ence texts with distinct surface forms have con- veyed the same meaning. Recent effort on the ap- plicability of these metrics reveals that while com- pelling text generation system ascend on …

Content truncated to 500 words