Using counterfactual contrast to improve compositional generalization for multi-step quantitative reasoning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14930–14943 July 9-14, 2023 ©2023 Association for Computational Linguistics Using counterfactual contrast to improve compositional generalization for multi-step quantitative reasoning Armineh Nourbakhsh Language Technologies Institute, Carnegie Mellon University J.P. Morgan AI Research anourbak@cs.cmu.edu Sameena Shah J.P. Morgan AI Research sameena.shah@jpmorgan.com Carolyn Rosé Language Technologies Institute, Carnegie Mellon University cprose@cs.cmu.edu Abstract In quantitative question answering, composi- tional generalization is one of the main chal- lenges of state of the art models, especially when longer sequences of reasoning steps are required. In this paper we propose Counter- Comp, a method that uses counterfactual sce- narios to generate samples with compositional contrast. Instead of a data augmentation ap- proach, CounterComp is based on metric learn- ing, which allows for direct sampling from the training set and circumvents the need for ad- ditional human labels. Our proposed auxiliary metric learning loss improves the performance of three state of the art models on four recently released datasets. We also show how the ap- proach can improve OOD performance on un- seen domains, as well as unseen compositions. Lastly, we demonstrate how the method can lead to better compositional attention patterns during training. 1 Introduction Enterprise documents such as reports, forms, and analytical articles often include quantitative data in tabular form. The data in these tables can be self-contained, but more commonly the surround- ing text provides more context that is necessary to understand the content. Answering questions over these hybrid tabular/text contexts requires reason- ing that combines verbal and quantitative seman- tics. Question answering over quantitative tabu- lar/text data has gained recent traction with the release of datasets such as FinQA (Chen et al., 2021b), TAT-QA (Zhu et al., 2021), and HiTab (Cheng et al., 2022). Table 2 shows an example of a question that requires quantitative reasoning to derive the answer. Given the question and the tabular context, the output is a single-step program that leads to the final answer of -20. A major challenge that state of the art models face is compositional generalization (Montague, # steps in output % wrong operator(s) % wrong operands % wrong order of operands 1 step 39.07 53.64 7.28 2 steps 46.75 46.75 6.50 3 steps 56.47 29.41 14.12 >= 4 steps 52.00 40.00 8.00 Table 1: Share of FinQANet errors due to the selection of wrong operators or operands when applied to the FinQA dataset (Chen et al., 2021b), broken down by the number of steps in the output. Note that the numbers are based on program accuracy, which accounts for each type of error separately, resulting in the each row summing up to 1. 1973), especially when the number of reasoning steps grows (Chen et al., 2021b). In the context of quantitative QA, compositional generalization refers to the model’s ability to generalize to new compositions of previously seen elements. As an example, if the model has encountered training ex- amples that demonstrate calculations for “growth rate” and “percent …

Content truncated to 500 words