Handling Divergent Reference Texts when Evaluating Table-to-Text Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4884–4895 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 4884 Handling Divergent Reference Texts when Evaluating Table-to-Text Generation Bhuwan Dhingra† ∗ Manaal Faruqui‡ Ankur Parikh‡ Ming-Wei Chang‡ Dipanjan Das‡ William W. Cohen†‡ † Carnegie Mellon University ‡ Google Research bdhingra@cs.cmu.edu {mfaruqui,aparikh,mingweichang,dipanjand,wcohen}@google.com Abstract Automatically constructed datasets for gener- ating text from semi-structured data (tables), such as WikiBio (Lebret et al., 2016), of- ten contain reference texts that diverge from the information in the corresponding semi- structured data. We show that metrics which rely solely on the reference texts, such as BLEU and ROUGE, show poor correlation with human judgments when those references diverge. We propose a new metric, PAR- ENT, which aligns n-grams from the refer- ence and generated texts to the semi-structured data before computing their precision and re- call. Through a large scale human evalua- tion study of table-to-text models for WikiBio, we show that PARENT correlates with human judgments better than existing text generation metrics. We also adapt and evaluate the infor- mation extraction based evaluation proposed in Wiseman et al. (2017), and show that PAR- ENT has comparable correlation to it, while being easier to use. We show that PARENT is also applicable when the reference texts are elicited from humans using the data from the WebNLG challenge.1 1 Introduction The task of generating natural language descrip- tions of structured data (such as tables) (Kukich, 1983; McKeown, 1985; Reiter and Dale, 1997) has seen a growth in interest with the rise of se- quence to sequence models that provide an easy way of encoding tables and generating text from them (Lebret et al., 2016; Wiseman et al., 2017; Novikova et al., 2017b; Gardent et al., 2017). For text generation tasks, the only gold standard metric is to show the output to humans for judg- ing its quality, but this is too expensive to apply ∗Work done during an internship at Google. 1Code and Data: http://www.cs.cmu.edu/ ~bdhingra/pages/parent.html repeatedly anytime small modifications are made to a system. Hence, automatic metrics that com- pare the generated text to one or more reference texts are routinely used to compare models (Ban- galore et al., 2000). For table-to-text generation, automatic evaluation has largely relied on BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004). The underlying assumption behind these metrics is that the reference text is gold-standard, i.e., it is the ideal target text that a system should gener- ate. In practice, however, when datasets are col- lected automatically and heuristically, the refer- ence texts are often not ideal. Figure 1 shows an example from the WikiBio dataset (Lebret et al., 2016). Here the reference contains extra informa- tion which no system can be expected to produce given only the associated table. We call such ref- erence texts divergent from the table. We show that existing automatic metrics, in- cluding BLEU, correlate poorly with human judg- ments when the evaluation sets contain divergent references …

Content truncated to 500 words