ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6279 - 6292 December 7-11, 2022 ©2022 Association for Computational Linguistics CONVFINQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering Zhiyu Chen1, Shiyang Li1, Charese Smiley2, Zhiqiang Ma2, Sameena Shah2 and William Yang Wang1 1University of California, Santa Barbara 2J.P. Morgan {zhiyuchen,shiyangli,william}@cs.ucsb.edu, {charese.h.smiley,zhiqiang.ma,sameena.shah}@jpmchase.com Abstract With the recent advance in large pre-trained language models, researchers have achieved record performances in NLP tasks that mostly focus on language pattern matching. The com- munity is experiencing the shift of the chal- lenge from how to model language to the im- itation of complex reasoning abilities like hu- man beings. In this work, we investigate the application domain of finance that involves real- world, complex numerical reasoning. We pro- pose a new large-scale dataset, CONVFINQA, aiming to study the chain of numerical reason- ing in conversational question answering. Our dataset poses great challenge in modeling long- range, complex numerical reasoning paths in real-world conversations. We conduct compre- hensive experiments and analyses with both the neural symbolic methods and the prompting- based methods, to provide insights into the rea- soning mechanisms of these two divisions. We believe our new dataset should serve as a valu- able resource to push forward the exploration of real-world, complex reasoning tasks as the next research focus. Our dataset and code is publicly available1. 1 Introduction The rapid advancement in developing large pre- trained language models (LM) has brought the nat- ural language processing research into a new era. Based on the well-known transformer (Vaswani et al., 2017) architecture, such large pre-trained LMs (Devlin et al., 2019; Radford et al., 2019; Raffel et al., 2020; Sanh et al., 2021; Wang et al., 2022) have set up new state-of-the-art results for many NLP tasks, with some of them approaching or even surpassing human performances, like on the SQuAD (Rajpurkar et al., 2016) dataset. We observe that the tasks with the essence of modeling language patterns can be well addressed by large pre-trained LMs. However, for the other kind of 1https://github.com/czyssrs/ConvFinQA 2010 2009 2008 share-based compensation cost $18.10 $14.60 $13.80 income tax benefit -$6.30 -$5.20 -$4.90 Financial report: … the total income tax benefit recognized for share-based compensation in the accompanying statements of income is also presented. Conversational QA: Q1: In the year of 2010, what was the share-based compensation cost? A1: 18.1 Q2: and what was the income tax benefit? A2: -6.3 Q3: what was, then, the sum of both? A3: add(18.1, -6.3) = 11.8 Q4: and what was that sum in 2009? A4: add(14.6, -5.2) = 9.4 Q5: what, then, was the change in the sum of those amounts from 2009 to 2010? A5: add(18.1, -6.3), add(14.6, -5.2), subtract(#0, #1) = 2.4 Figure 1: An example from CONVFINQA: each question may depend on previous questions to answer. tasks requiring complex reasoning abilities, current researches are still away from satisfactory perfor- mances (Wei et al., 2022). Traditional methods on reasoning tasks typically take neural symbolic …

Content truncated to 500 words