Why We Need New Evaluation Metrics for NLG

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2241–2252 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Why We Need New Evaluation Metrics for NLG Jekaterina Novikova, Ondˇrej Duˇsek, Amanda Cercas Curry and Verena Rieser School of Mathematical and Computer Sciences Heriot-Watt University, Edinburgh j.novikova, o.dusek, ac293, v.t.rieser@hw.ac.uk Abstract The majority of NLG evaluation relies on automatic metrics, such as BLEU. In this paper, we motivate the need for novel, system- and data-independent au- tomatic evaluation methods: We inves- tigate a wide range of metrics, includ- ing state-of-the-art word-based and novel grammar-based ones, and demonstrate that they only weakly reflect human judge- ments of system outputs as generated by data-driven, end-to-end NLG. We also show that metric performance is data- and system-specific. Nevertheless, our results also suggest that automatic metrics per- form reliably at system-level and can sup- port system development by finding cases where a system performs poorly. 1 Introduction Automatic evaluation measures, such as BLEU (Pa- pineni et al., 2002), are used with increasing fre- quency to evaluate Natural Language Generation (NLG) systems: Up to 60% of NLG research published between 2012–2015 relies on automatic metrics (Gkatzia and Mahamood, 2015). Auto- matic evaluation is popular because it is cheaper and faster to run than human evaluation, and it is needed for automatic benchmarking and tuning of algorithms. The use of such metrics is, however, only sensible if they are known to be sufficiently correlated with human preferences. This is rarely the case, as shown by various studies in NLG (Stent et al., 2005; Belz and Reiter, 2006; Reiter and Belz, 2009), as well as in related fields, such as dialogue systems (Liu et al., 2016), machine translation (MT) (Callison-Burch et al., 2006), and image captioning (Elliott and Keller, 2014; Kilick- aya et al., 2017). This paper follows on from the above previous work and presents another evalu- ation study into automatic metrics with the aim to firmly establish the need for new metrics. We consider this paper to be the most complete study to date, across metrics, systems, datasets and do- mains, focusing on recent advances in data-driven NLG. In contrast to previous work, we are the first to: • Target end-to-end data-driven NLG, where we compare 3 different approaches. In contrast to NLG methods evaluated in previous work, our sys- tems can produce ungrammatical output by (a) generating word-by-word, and (b) learning from noisy data. • Compare a large number of 21 automated met- rics, including novel grammar-based ones. • Report results on two different domains and three different datasets, which allows us to draw more general conclusions. • Conduct a detailed error analysis, which sug- gests that, while metrics can be reasonable indi- cators at the system-level, they are not reliable at the sentence-level. • Make all associated code and data publicly avail- able, including detailed analysis results.1 2 End-to-End NLG Systems In this paper, we focus on recent end-to-end, data- driven NLG methods, which jointly learn sentence planning …

Content truncated to 500 words