ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

Findings of the Association for Computational Linguistics: ACL 2022, pages 2263 - 2279 May 22-27, 2022 c⃝2022 Association for Computational Linguistics ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning Ahmed Masry♣, Do Xuan Long♠, Jia Qing Tan♠, Shafiq Joty♠♦, Enamul Hoque♣ ♣York University, Canada ♠Nanyang Technological University, Singapore, ♦Salesforce Research ♣{masry20, enamulh}@yorku.ca ♠{xuanlong001@e.ntu, C190022@e.ntu, srjoty@ntu}.edu.sg Abstract Charts are very popular for analyzing data. When exploring charts, people often ask a va- riety of complex reasoning questions that in- volve several logical and arithmetic operations. They also commonly refer to visual features of a chart in their questions. However, most existing datasets do not focus on such com- plex reasoning questions as their questions are template-based and answers come from a fixed- vocabulary. In this work, we present a large- scale benchmark covering 9.6K human-written questions as well as 23.1K questions gener- ated from human-written chart summaries. To address the unique challenges in our bench- mark involving visual and logical reasoning over charts, we present two transformer-based models that combine visual features and the data table of the chart in a unified way to an- swer questions. While our models achieve the state-of-the-art results on the previous datasets as well as on our benchmark, the evaluation also reveals several challenges in answering complex reasoning questions. 1 Introduction Data visualizations such as bar charts and line charts have become popular in analyzing data and making informed decisions. To analyze data, often people ask complex reasoning questions about charts involving arithmetic and logical opera- tions (Kim et al., 2020). Answering such questions requires a significant amount of perceptual and cog- nitive efforts as people need to combine multiple operations such as retrieving values, comparing values, finding maximum, calculating sums and dif- ferences of values. For example, the question Q1 in Fig. 1 requires the user to compute the differences between the two lines for each year and find the year with the highest difference. The goal of a Chart Question Answering (ChartQA) system is to help users by taking a chart and a natural language question as input and pre- Q1: Which year has the most divergent opinions about Brazil’s economy? Answer: 2015 Q2: What is the peak value of the orange line? Answer: 87 Figure 1: Sample questions in our benchmark. dicting the answer. This task differs from other QA tasks such as QA on texts (Rajpurkar et al., 2016) and tables (Pasupat and Liang, 2015) because the input for ChartQA is a visual representation of data that can draw a reader’s attention to various promi- nent features such as trends and outliers (Kim et al., 2020, 2021). Also, people tend to ask questions by referring to visual attributes of marks. For example, in Fig. 1, Q2 refers to the color of a mark (‘line’) and its attribute (‘peak’) in the chart. While the task of ChartQA has received growing attentions in recent years, existing datasets have several major limitations: (i) the questions are gen- erated automatically …

Content truncated to 500 words