Chart-to-Text: A Large-Scale Benchmark for Chart Summarization

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4005 - 4023 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Chart-to-Text: A Large-Scale Benchmark for Chart Summarization Shankar Kantharaj♣∗, Rixie Tiffany Ko Leong♦∗, Xiang Lin♦∗, Ahmed Masry♣∗ Megh Thakkar♦∗, Enamul Hoque♣, Shafiq Joty♦♠ ♣York University, Canada, ♦Nanyang Technological University, Singapore ♠Salesforce Research Asia, Singapore {shankark, masry20, enamulh}@yorku.ca {rleong007, linx0057, srjoty}@ntu.edu.sg megh.1211@gmail.com Abstract Charts are commonly used for exploring data and communicating insights. Generating nat- ural language summaries from charts can be very helpful for people in inferring key insights that would otherwise require a lot of cognitive and perceptual efforts. We present Chart-to- text, a large-scale benchmark with two datasets and a total of 44,096 charts covering a wide range of topics and chart types. We explain the dataset construction process and analyze the datasets. We also introduce a number of state- of-the-art neural models as baselines that utilize image captioning and data-to-text generation techniques to tackle two problem variations: one assumes the underlying data table of the chart is available while the other needs to ex- tract data from chart images. Our analysis with automatic and human evaluation shows that while our best models usually generate fluent summaries and yield reasonable BLEU scores, they also suffer from hallucinations and factual errors as well as difficulties in correctly explain- ing complex patterns and trends in charts. 1 Introduction Data visualizations such as bar charts, line charts, and pie charts are very popular for presenting quan- titative data. Often people use such charts to get important insights from data and make informed decisions. However, it is well-known that inferring key insights from the charts can be quite challeng- ing and time-consuming, as it may require a lot of cognitive and perceptual efforts (Pérez-Echeverría et al., 2018; Whitaker and Jacobbe, 2017). Automatic chart summarization is a task where the goal is to explain a chart and summarize key takeaways from it in natural language. Chart sum- marization has several key benefits and potential applications. First, chart summaries can help peo- ple identify key insights from charts that they might ∗Equal contribution. Listing order is based on the alpha- betical ordering of author surnames. Gold: In 2019, Singapore imported approximately 236.8 billion Singapore dollars worth of machinery and equipment, making it the country’s largest import commodity by value. This was followed by the import of mineral fuels and lubricants, valued at 102.7 billion Singapore dollars. TAB-T5: Machinery and equipment was the most valuable commodity for Singapore in 2019, with an import value of 236.8 billion Singapore dollars. Mineral fuels and lubricants were the second most valuable commodity for Singapore, with an import value of 102.7 billion Singapore dollars. Figure 1: An example chart-summary pair from our Benchmark and the output from one of the best models (TAB-T5). have missed otherwise. In a study on a chart corpus, Carberry et al. (2006) found that chart authors often failed to convey key insights from charts in their …

Content truncated to 500 words