STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 3275–3284, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 3275 STL-CQA: Structure-based Transformers with Localization and Encoding for Chart Question Answering Hrituraj Singh Adobe Research hrisingh@adobe.com Sumit Shekhar Adobe Research sushekha@adobe.com Abstract Chart Question Answering (CQA) is the task of answering natural language questions about visualisations in the chart image. Recent so- lutions, inspired by VQA approaches, rely on image-based attention for question/answering while ignoring the inherent chart structure. We propose STL-CQA which improves the ques- tion/answering through sequential elements lo- calization, question encoding and then, a struc- tural transformer-based learning approach. We conduct extensive experiments while propos- ing pre-training tasks, methodology and also an improved dataset with more complex and balanced questions of different types. The proposed methodology shows a significant ac- curacy improvement compared to the state- of-the-art approaches on various chart Q/A datasets, while outperforming even human baseline on the DVQA Dataset. We also demonstrate interpretability while examining different components in the inference pipeline. 1 Introduction Charts Question Answering (CQA) (Kafle et al., 2018; Kahou et al., 2017; Chaudhry et al., 2020; Methani et al., 2020a) is the task designed on the lines of Visual Question Answering (VQA) (Antol et al., 2015; Malinowski and Fritz, 2014) which re- quires answering natural language questions about the data visualisations such as bar charts, pie charts, etc. The problem provides us with ability to under- stand charts using natural language queries, as well as grounding to the natural language statements for the reasoning operations being carried out to retrieve the final answer to the query. CQA is a challenging task because of the follow- ing reasons - (a) large question/answer vocabulary due to chart-specific words, (b) Requirements of multi-modal fine-grained reasoning through under- standing of natural language question as well as the visualizations. This is different from VQA, where the answer dictionary is typically limited, and the reasoning is coarse-grained as compared to that required for data visualisations, where finer details like bar length and color can heavily influence both the reasoning and the answer. Despite data visualisations being ubiquitous in documents, the problem has received sparse at- tention in the literature. The earlier datasets like DVQA (Kafle et al., 2018) and FigureQA (Ka- hou et al., 2017) consist of charts generated from synthetic data, though there has been a push for data charts generated from real sources (Chaudhry et al., 2020; Methani et al., 2020a) as well. Due to the problems discussed above, the prior work noted that VQA algorithms cannot be applied di- rectly to CQA. Hence, different CQA methods in- troduce modifications for the problem, while build- ing on the backbone of VQA approaches. While FigureQA (Kahou et al., 2017) uses relational networks for question/answering, DVQA (Kafle et al., 2018) combines text detection and VQA- based attention modules to answer chart questions. LEAF-QA (Chaudhry et al., 2020) encodes ques- tion/answers in terms of chart elements, to handle infinite vocabulary problem, while resorting to a …

Content truncated to 500 words