FinQA: A Dataset of Numerical Reasoning over Financial Data

FINQA: A Dataset of Numerical Reasoning over Financial Data Zhiyu Chen1, Wenhu Chen1, Charese Smiley2, Sameena Shah2, Iana Borova1, Dylan Langdon1, Reema Moussa1, Matt Beane1, Ting-Hao Huang3, Bryan Routledge4 and William Yang Wang1 1University of California, Santa Barbara 2J.P. Morgan 3Pennsylvania State University 4Carnegie Mellon University {zhiyuchen,william}@cs.ucsb.edu Abstract The sheer volume of financial statements makes it difficult for humans to access and an- alyze a business’s financials. Robust numeri- cal reasoning likewise faces unique challenges in this domain. In this work, we focus on answering deep questions over financial data, aiming to automate the analysis of a large cor- pus of financial documents. In contrast to ex- isting tasks on general domain, the finance do- main includes complex numerical reasoning and understanding of heterogeneous represen- tations. To facilitate analytical progress, we propose a new large-scale dataset, FINQA, with Question-Answering pairs over Financial reports, written by financial experts. We also annotate the gold reasoning programs to en- sure full explainability. We further introduce baselines and conduct comprehensive experi- ments in our dataset. The results demonstrate that popular, large, pre-trained models fall far short of expert humans in acquiring finance knowledge and in complex multi-step numer- ical reasoning on that knowledge. Our dataset — the first of its kind — should therefore en- able significant, new community research into complex application domains. The dataset and code are publicly available1. 1 Introduction Financial analysis is a critical means of assessing business performance, and the consequences of poor analysis can involve costs of billions of dol- lars (Jerven, 2013; MacKenzie, 2008). To facilitate high quality, timely decision making, profession- als — such as analysts or investors — perform complex quantitative analysis to select informa- tion from financial reports. Such analysis demands advanced expertise in reasoning among heteroge- neous (structured and unstructured) data sources and performing complex numerical reasoning, for example, comparing financial ratios of profitabil- ity or growth. These challenges are compounded 1https://github.com/czyssrs/FinQA by an exponentially expanding collection of com- pany financial documents (MacKenzie et al., 2012; Lange et al., 2016) such that it is genuinely unclear whether dedicated human effort can produce fiscal analysis of sufficient quality for current decision making. This poses an interesting question: can we automate such deep analysis of financial data? A few NLP studies in Question Answering (QA) explored the numerical reasoning capabilities needed to answer questions correctly. For exam- ple, the DROP dataset (Dua et al., 2019) focused on Wikipedia-based questions that require numer- ical reasoning, e.g., “Where did Charles travel to first, Castile or Barcelona?” needs a comparison between the times of two events. However, most prior work only targeted the general domain, where the questions involve much less calculation (mostly one-step calculation) than that of the financial do- main. Financial QA is more challenging than clas- sic QA (Rajpurkar et al., 2018; Yang et al., 2018) because it requires the system to spot relevant in- formation across heterogeneous sources, such as tables and unstructured texts, and then create a numerical reasoning path to connect all …

Content truncated to 500 words