VisText: A Benchmark for Semantically Rich Chart Captioning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 7268–7298 July 9-14, 2023 ©2023 Association for Computational Linguistics VisText: A Benchmark for Semantically Rich Chart Captioning Benny J. Tang∗ MIT CSAIL benjtang@csail.mit.edu Angie Boggust∗ MIT CSAIL aboggust@csail.mit.edu Arvind Satyanarayan MIT CSAIL arvindsatya@mit.edu Abstract Captions that describe or explain charts help im- prove recall and comprehension of the depicted data and provide a more accessible medium for people with visual disabilities. However, cur- rent approaches for automatically generating such captions struggle to articulate the percep- tual or cognitive features that are the hallmark of charts (e.g., complex trends and patterns). In response, we introduce VisText: a dataset of 12,441 pairs of charts and captions that describe the charts’ construction, report key statistics, and identify perceptual and cognitive phenom- ena. In VisText, a chart is available as three rep- resentations: a rasterized image, a backing data table, and a scene graph — a hierarchical rep- resentation of a chart’s visual elements akin to a web page’s Document Object Model (DOM). To evaluate the impact of VisText, we fine-tune state-of-the-art language models on our chart captioning task and apply prefix-tuning to pro- duce captions that vary the semantic content they convey. Our models generate coherent, semantically rich captions and perform on par with state-of-the-art chart captioning models across machine translation and text generation metrics. Through qualitative analysis, we iden- tify six broad categories of errors that our mod- els make that can inform future work. 1 Introduction Studies have shown that captions can improve the recall and comprehension of the data that charts depict (Hegarty and Just, 1993; Large et al., 1995). For instance, when a caption emphasizes visually prominent features of a chart, like a peak or a sharply declining trend, readers treat this infor- mation as the key takeaway (Kim et al., 2021). Moreover, for people with visual disabilities, cap- tions (or equivalent descriptions such as alt text) are often the only means of accessing the presented data. However, as evidenced by numerous guide- lines (Jung et al., 2021), producing high-quality *Both authors contributed equally to this work. chart captions is a non-trivial and laborious manual process. Thus, despite these advantages, charts are only rarely captioned in practice (Lundgard and Satyanarayan, 2022). To bridge this gap, several research communi- ties have begun to explore methods for automat- ically generating chart captions, including using templates and heuristics (Demir et al., 2008; Srini- vasan et al., 2019), adapting image captioning tech- niques (Balaji et al., 2018; Chen et al., 2019a), or via data-to-text machine translation (Kantharaj et al., 2022; Obeid and Hoque, 2020). While promising, these approaches have largely produced captions that either describe a chart’s construction (e.g., “The graph is plot between ’Number of peo- ple’ x-axis over ’Movie Genres’ y-axis” (Balaji et al., 2018)) or provide statistical summaries (e.g., “Machinery and equipment was the most valuable commodity for Singapore in 2019” (Kantharaj et al., 2022)). However, these captions do not articulate the perceptual …

Content truncated to 500 words