SemEval-2019 Task 10: Math Question Answering

Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), pages 893–899 Minneapolis, Minnesota, USA, June 6–7, 2019. ©2019 Association for Computational Linguistics 893 SemEval 2019 Task 10: Math Question Answering Mark Hopkins Department of Mathematics Reed College hopkinsm@reed.edu Ronan Le Bras and Cristian Petrescu-Prahova Allen Institute for Artificial Intelligence {ronanlb,cristipp}@allenai.org Gabriel Stanovsky and Hannaneh Hajishirzi and Rik Koncel-Kedziorski Allen School of Computer Science and Engineering University of Washington {gabis, hannaneh,kedzior}@uw.edu Abstract We report on the SemEval 2019 task on math question answering. We provided a question set derived from Math SAT practice exams, in- cluding 2778 training questions and 1082 test questions. For a significant subset of these questions, we also provided SMT-LIB logical form annotations and an interpreter that could solve these logical forms. Systems were eval- uated based on the percentage of correctly an- swered questions. The top system correctly answered 45% of the test questions, a con- siderable improvement over the 17% random guessing baseline. 1 Overview Over the past four years, there has been a surge of interest in math question answering. Research groups from around the globe have published pa- pers on the topic, including MIT (Kushman et al., 2014), University of Washington (Hosseini et al., 2014; Koncel-Kedziorski et al., 2015), National Institute of Informatics, Japan (Matsuzaki et al., 2014), University of Illinois (Roy and Roth, 2015), Microsoft Research (Shi et al., 2015; Upadhyay and Chang, 2016), Baidu (Zhou et al., 2015), Ari- zona State University (Mitra and Baral, 2016), KU Leuven (Dries et al., 2017), Carnegie Mellon Uni- versity (Sachan et al., 2017), Tencent (Wang et al., 2017), and DeepMind (Ling et al., 2017). Math question answering has several attractive properties that have rekindled this interest: 1. It is easy to evaluate. Usually there is a sin- gle correct answer for a given question, either numeric or multiple-choice. 2. In order to achieve robust results, systems require some (explicit or implicit) seman- tic representation of the question language. Consider the first question in Figure 1. The Closed-vocabulary algebra: Suppose 3x + y = 15, where x is a positive integer. What is the difference between the largest possible value of y and the smallest possible value of x, assuming that y is also a positive integer? Open-vocabulary algebra: At a basketball tournament involving 8 teams, each team played 4 games with each of the other teams. How many games were played at this tourna- ment? Geometry: The lengths of two sides of a tri- angle are (x − 2) and (x + 2), where x > 2. Which of the following ranges includes all and only the possible values of the third side y? (A) 0 < y < x (B) 0 < y < 2x (C) 4 < y < 2x Figure 1: Example math questions from different gen- res. correct answer (11) has a subtle relationship to the other quantities mentioned in the text (3 and 15). There is no obvious shortcut (like word association metrics on a bag-of-words representation of …

Content truncated to 500 words