MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6588 - 6600 May 22-27, 2022 c⃝2022 Association for Computational Linguistics MULTIHIERTT: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data Yilun Zhao1 Yunxiang Li2 Chenying Li3 Rui Zhang4 1Yale University 2The Chinese University of Hong Kong 3Northeastern University 4Penn State University yilun.zhao@yale.edu 1155124348@link.cuhk.edu.hk li.chenyin@northeastern.edu rmz5227@psu.edu Abstract Numerical reasoning over hybrid data contain- ing both textual and tabular content (e.g., finan- cial reports) has recently attracted much atten- tion in the NLP community. However, existing question answering (QA) benchmarks over hy- brid data only include a single flat table in each document and thus lack examples of multi- step numerical reasoning across multiple hi- erarchical tables. To facilitate data analyti- cal progress, we construct a new large-scale benchmark, MULTIHIERTT, with QA pairs over Multi Hierarchical Tabular and Textual data. MULTIHIERTT is built from a wealth of financial reports and has the following unique characteristics: 1) each document contain mul- tiple tables and longer unstructured texts; 2) most of tables contained are hierarchical; 3) the reasoning process required for each ques- tion is more complex and challenging than ex- isting benchmarks; and 4) fine-grained anno- tations of reasoning processes and supporting facts are provided to reveal complex numeri- cal reasoning. We further introduce a novel QA model termed MT2Net, which first applies facts retrieving to extract relevant supporting facts from both tables and text and then uses a reasoning module to perform symbolic reason- ing over retrieved facts. We conduct compre- hensive experiments on various baselines. The experimental results show that MULTIHIERTT presents a strong challenge for existing base- lines whose results lag far behind the perfor- mance of human experts. The dataset and code are publicly available at https://github. com/psunlpgroup/MultiHiertt. 1 Introduction In recent years, as key to many NLP tasks such as QA, there is a flurry of works on numerical rea- soning over various types of data including textual data (Dua et al., 2019; Amini et al., 2019; Xie and Sun, 2019) and tabular data (Moosavi et al., 2021; Suadaa et al., 2021). More recently, numerical Figure 1: An example of MULTIHIERTT: The system needs to first locate which segment got the most funds in 2017 in the second hierarchical table, then select rel- evant numbers from the first hierarchical table and gen- erate the correct reasoning program to get the answer. The annotated supporting facts are highlighted in red, and the hierarchical column and row headers are high- lighted in orange and green, respectively. reasoning over hybrid data containing both textual and tabular content (Zhu et al., 2021; Chen et al., 2021) has attracted much attention. For example, 6588 the FinQA dataset (Chen et al., 2021) focuses on questions that require numerical reasoning over fi- nancial report pages, e.g., "What portion of the total identifiable net assets is in cash?". Such questions need the system to locate relevant cells in the tabu- lar content and then perform a division operation …

Content truncated to 500 words