TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3277–3287 August 1–6, 2021. ©2021 Association for Computational Linguistics 3277 TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance Fengbin Zhu1,2, Wenqiang Lei1∗, Youcheng Huang 3, Chao Wang2, Shuo Zhang4, Jiancheng Lv3, Fuli Feng1, Tat-Seng Chua1 1National University of Singapore, 26Estates Pte Ltd, 3Sichuan University, 4Bloomberg {zhfengbin, wenqianglei}@gmail.com, wangchao@6estates.com Abstract Hybrid data combining both tabular and tex- tual content (e.g., financial reports) are quite pervasive in the real world. However, Ques- tion Answering (QA) over such hybrid data is largely neglected in existing research. In this work, we extract samples from real financial reports to build a new large-scale QA dataset containing both Tabular And Textual data, named TAT-QA, where numerical reasoning is usually required to infer the answer, such as addition, subtraction, multiplication, divi- sion, counting, comparison/sorting, and their compositions. We further propose a novel QA model termed TAGOP, which is capable of rea- soning over both tables and text. It adopts se- quence tagging to extract relevant cells from the table along with relevant spans from the text to infer their semantics, and then applies symbolic reasoning over them with a set of aggregation operators to arrive at the final an- swer. TAGOP achieves 58.0% in F1, which is an 11.1% absolute increase over the pre- vious best baseline model, according to our experiments on TAT-QA. But this result still lags far behind the performance of human expert, i.e. 90.8% in F1. It demonstrates that our TAT-QA is very challenging and can serve as a benchmark for training and test- ing powerful QA models that address hybrid data. Our dataset is publicly available for non- commercial use at https://nextplusplus. github.io/TAT-QA/. 1 Introduction Existing QA systems largely focus on only unstruc- tured text (Hermann et al., 2015; Rajpurkar et al., 2016; Dua et al., 2019; Yang et al., 2018; Li et al., 2020; Nie et al., 2020), structured knowledge base (KB) (Berant et al., 2013; Yih et al., 2015; Talmor and Berant, 2018), or semi-structured tables (Pasu- pat and Liang, 2015; Zhong et al., 2017; Yu et al., ∗Corresponding author 2018; Zhang and Balog, 2019; Zhang et al., 2020). Though receiving growing interests (Das et al., 2017; Sun et al., 2019; Chen et al., 2020b, 2021), works on hybrid data comprising of unstructured text and structured or semi-structured KB/tables are rare. Recently, Chen et al. (2020b) attempt to simulate a type of hybrid data through manually linking table cells to Wiki pages via hyperlinks. However, such connection between table and text is relatively loose. In the real world, a more common hybrid data form is, the table (that usually contains numbers) is more comprehensively linked to text, e.g., se- mantically related or complementary. Such hybrid data are very pervasive in various scenarios like scientific research papers, medical reports, finan- cial reports, etc. The left box of Figure 1 shows a …

Content truncated to 500 words