VisToT: Vision-Augmented Table-to-Text Generation

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9936 - 9949 December 7-11, 2022 ©2022 Association for Computational Linguistics VISTOT: Vision-Augmented Table-to-Text Generation Prajwal Gatti1, Anand Mishra1, Manish Gupta2, Mithun Das Gupta2 1Indian Institute of Technology Jodhpur 2Microsoft {pgatti,mishra}@iitj.ac.in, {gmanish,migupta}@microsoft.com Abstract Table-to-text generation has been widely stud- ied in the Natural Language Processing com- munity in the recent years. We give a new perspective to this problem by incorporating signals from both tables as well as associated images to generate relevant text. While tables contain a structured list of facts, images are a rich source of unstructured visual information. For example, in the tourism domain, images can be used to infer knowledge such as the type of landmark (e.g., church), its architecture (e.g., Ancient Roman), and composition (e.g., white marble). Therefore, in this paper, we introduce the novel task of Vision-augmented Table-To- Text Generation (VISTOT), defined as follows: given a table and an associated image, produce a descriptive sentence conditioned on the mul- timodal input. For the task, we present a novel multimodal table-to-text dataset, WIKILAND- MARKS, covering 73,084 unique world land- marks. Further, we also present a competitive architecture, namely, VT3 that generates accu- rate sentences conditioned on the image and table pairs. Through extensive analyses and experiments, we show that visual cues from im- ages are helpful in (i) inferring missing informa- tion from incomplete or sparse tables, and (ii) strengthening the importance of useful infor- mation from noisy tables for natural language generation. We make the code and data pub- licly available1. 1 Introduction Structured data-to-text generation is a well-studied problem that demands a model to produce mean- ingful and factually accurate sentences based on its comprehension of the source context that are usually in the form of fact graphs, tables, or hierar- chical spreadsheets (Lebret et al., 2016; Wiseman et al., 2017). Many datasets and several competitive 1https://vl2g.github.io/projects/vistot Conventional Transformer-based Table-to-Text: “Lough Leane is a 4700 acre estate in Killarney, County Kerry, Ireland.” Vision-augmented Table-to-Text (this work): “Lough Leane is a large lake in Kil- larney, County Kerry, Ireland.” Ground Truth: “Lough Leane is the largest of the three lakes of Killarney, in County Kerry.” Figure 1: VISTOT: Vision-augmented Table-to-Text. In this novel proposed task, the goal is to explore the utility of visual cues besides the information in the table for natural language generation. Note the presence of large lake in the text generated by our proposed model; on the other hand, existing table-to-text models wrongly generate estate in the output text. models have been proposed for data-to-text gener- ation in the last few years. Data-to-text has been used to generate variety of texts including weather reports (Angeli et al., 2010), sports news (Wiseman et al., 2017) and biographies (Lebret et al., 2016). Specifically, for table-to-text, several bench- mark datasets have been proposed including Wik- iBio (Lebret et al., 2016), WebNLG (Gardent et al., 2017), E2E (Novikova et al., 2017), DART (Nan et al., 2021), RotoWire (Wiseman et al., 2017), …

Content truncated to 500 words