GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 513–523 August 1–6, 2021. ©2021 Association for Computational Linguistics 513 GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoning Jiaqi Chen1∗, Jianheng Tang2∗, Jinghui Qin1, Xiaodan Liang2†, Lingbo Liu1,3, Eric P. Xing4, Liang Lin1,3 1Sun Yat-sen University, 2Shenzhen Campus of Sun Yat-sen University, 3Dark Matter AI Inc., 4Mohamed bin Zayed University of Artificial Intelligence {jadgechen,sqrt3tjh,xdliang328,liulingbo918}@gmail.com, qinjingh@mail2.sysu.edu.cn, eric.xing@petuum.com, linliang@ieee.org Abstract Automatic math problem solving has re- cently attracted increasing attention as a long- standing AI benchmark. In this paper, we fo- cus on solving geometric problems, which re- quires a comprehensive understanding of tex- tual descriptions, visual diagrams, and theo- rem knowledge. However, the existing meth- ods were highly dependent on handcraft rules and were merely evaluated on small-scale datasets. Therefore, we propose a Geometric Question Answering dataset GeoQA, con- taining 5,010 geometric problems with cor- responding annotated programs, which illus- trate the solving process of the given prob- lems. Compared with another publicly avail- able dataset GeoS, GeoQA is 25 times larger, in which the program annotations can pro- vide a practical testbed for future research on explicit and explainable numerical reasoning. Moreover, we introduce a Neural Geometric Solver (NGS) to address geometric problems by comprehensively parsing multimodal infor- mation and generating interpretable programs. We further add multiple self-supervised auxil- iary tasks on NGS to enhance cross-modal se- mantic representation. Extensive experiments on GeoQA validate the effectiveness of our proposed NGS and auxiliary tasks. How- ever, the results are still significantly lower than human performance, which leaves large room for future research. Our benchmark and code are released at https://github.com/chen- judge/GeoQA. 1 Introduction In recent years, developing machine learning sys- tems to solve math word problems (MWPs) auto- matically has attracted increasing attention due to its high academic value and the great application potential in smart education (Bajaj and Sharma, ∗Equal contribution. †Corresponding author. As shown in the figure, in �O, AB is the chord, OC�AB, if the radius of �O is 5 (N0) and CE=2 (N1), then the length of AB is () A. 2 B. 4 C. 6 D. 8 Answer: D. 8 Problem Type: Length Calculation Knowledge Points: Vertical Diameter, Pythagorean Theorem Problem Solving Explanations: OE=OC-CE=5-2=3. According to the Pythagorean Theorem, AE = ��� � ���= �� � ��= 4. Thus, AB=2AE=8. Annotated Programs: Minus | N0 | N1 | PythagoreanMinus | N0 | V0 | Double | V1 Step1: Minus(N0, N1) = 5 – 2 = 3 (V0) Step2: PythagoreanMinus(N0, V0) = �� � �� = 4 (V1) Step3: Double(V1) = 2�4 = 8 (V2) Figure 1: Illustration of a typical geometry problem with the annotated programs in our GeoQA dataset. 2018; Lin et al., 2018). Most of the existing methods focus on solving arithmetic and algebraic problems, including traditional machine learning approaches (Kushman et al., 2014; Zhou et al., 2015; Huang et al., 2016) and network-based mod- els (Wang et al., 2017, 2018; Xie and Sun, 2019), while solving geometric problems has been rarely …

Content truncated to 500 words