Improving compositional generalization for multi-step quantitative reasoning in question answering

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1916 - 1932 December 7-11, 2022 ©2022 Association for Computational Linguistics Improving compositional generalization for multi-step quantitative reasoning in question answering Armineh Nourbakhsh1,2, Cathy Jiao1, Sameena Shah2, Carolyn P. Rosé1 1Language Technologies Institute, Carnegie Mellon University 2J.P. Morgan AI Research anourbak@cs.cmu.edu Abstract Quantitative reasoning is an important aspect of question answering, especially when numeric and verbal cues interact to indicate sophisti- cated, multi-step programs. In this paper, we demonstrate how modeling the compositional nature of quantitative text can enhance the per- formance and robustness of QA models, allow- ing them to capture arithmetic logic that is ex- pressed verbally. Borrowing from the literature on semantic parsing, we propose a method that encourages the QA models to adjust their at- tention patterns and capture input/output align- ments that are meaningful to the reasoning task. We show how this strategy improves program accuracy and renders the models more robust against overfitting as the number of reasoning steps grows. Our approach is designed as a standalone module which can be prepended to many existing models and trained in an end- to-end fashion without the need for additional supervisory signal. As part of this exercise, we also create a unified dataset building on four previously released numerical QA datasets over tabular data 1. 1 Introduction Any natural language system that processes or in- teracts with numeric data requires quantitative rea- soning to function. This has inspired research in several NLP domains, including reading compre- hension (Andor et al., 2019; Mishra et al., 2022), textual entailment (Roy et al., 2015; Ravichander et al., 2019), data-to-text generation (Parikh et al., 2020; Suadaa et al., 2021), and question answering (Chen et al., 2020; Zhang et al., 2021). A major challenge in quantitative reasoning is the interplay between numeric expressions and natural language (Roy et al., 2015). Standard neural approaches rely heavily on lexical matching, leading to overfitting over spurious verbal patterns. In contrast, a purely 1The dataset and code are available at https://github. com/ArmiNouri/CompAQT symbolic approach excels at numerical reasoning, but struggles when sophisticated verbal reasoning is required (Ravichander et al., 2019). In this pa- per we introduce a novel attention strategy that captures the interplay between numeric and verbal modalities, which improves program accuracy and renders models more robust to overfitting. Focusing on the question answering task, we show how our proposed method, named Com- pAQT (COMPositional Attention for QuanTitative reasoning), enables the model to attend to relevant parts of text at each reasoning step. CompAQT enhances the performance of state of the art models on several recently released QA datasets, especially for multi-step programs. It is implemented as a plug-and-play module that can be added to existing models with minimal effort and without the need for any additional supervision. Concretely, we offer the following contributions: • We propose a compositional attention module equipped with an alignment loss that improves SOTA performance on numeric QA tasks. • We demonstrate how the proposed approach …

Content truncated to 500 words