Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8029–8036 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8029 Data-QuestEval: A Reference-less Metric for Data-to-Text Semantic Evaluation Clément Rebuffel ∗ 1,2, Thomas Scialom ∗ 1,3, Laure Soulier1, Benjamin Piwowarski1, Sylvain Lamprier1, Jacopo Staiano3, Geoffrey Scoutheeten2, Patrick Gallinari1,4 1 Sorbonne Université, CNRS, LIP6, F-75005 Paris, France 2 BNP Paribas, Paris, France 3 reciTAL, Paris, France 4 Criteo AI Lab, Paris, France Abstract QUESTEVAL is a reference-less metric used in text-to-text tasks, that compares the gen- erated summaries directly to the source text, by automatically asking and answering ques- tions. Its adaptation to Data-to-Text tasks is not straightforward as it requires multimodal Question Generation and Answering systems on the considered tasks, which are seldom available. To this purpose, we propose a method to build synthetic multimodal corpora enabling to train multimodal components for a data-QuestEval metric. The resulting met- ric is reference-less and multimodal; it obtains state-of-the-art correlations with human judg- ment on the WebNLG and WikiBio bench- marks. We make data-QUESTEVAL’s code and models available for reproducibility pur- pose, as part of the QUESTEVAL project.1 1 Introduction Data-to-Text Generation (DTG) aims at generating descriptions in natural language given a structured input, e.g. a table (Gatt and Krahmer, 2018). Relia- bility and precision of generated texts is currently regarded as a major issue in DTG (Narayan and Gardent, 2020), with experimental surveys show- ing that real-life end users of DTG systems care more about accuracy than about readability (Reiter and Belz, 2009). Neural NLG systems are known to be fluent, but prone to hallucinations (Lee et al., 2018), i.e. they tend to include nonfactual informa- tion. However, their evaluation remains an open research problem (Novikova et al., 2017). A recent approach, QuestEval (Scialom et al., 2021) has shown significant improvement over stan- dard metrics on Summarization tasks. To measure semantic matching between an evaluated summary and its source document, QuestEval relies on Ques- tion Generation and Answering (QG/QA) systems. ∗Equal contribution 1https://github.com/ThomasScialom/QuestEval QuestEval + Recall QG QA Questions (Qx) Scores ( eq.2 ) QuestEval scores Table Table description Precision QG QA Questions (Qy) Scores ( eq.1 ) Table Table description Figure 1: Data-QUESTEVAL Flowchart. Figure adapted from the work of Scialom et al. (2021) (equa- tion numbers refer to equations in the original paper). As illustrated in Figure 1, a QG system generates a set of relevant questions conditioned on the source document, which are then asked on its generated summary. Conversely, questions generated from the summary are answered using only the source input. If the answers provided by the QA systems are correct, the summary is deemed consistent with its source document. Can QuestEval be adapted for evaluation on DTG tasks? So far, QuestEval’s QG/QA sys- tems have been trained on a purely textual dataset, SQuAD (Rajpurkar et al., 2016), which restricts the evaluation to comparisons between two texts. Unfortunately, DTG inputs are of different modali- ties than text (e.g. structured tables). In the absence of specific …

Content truncated to 500 words