Is a Question Decomposition Unit All We Need?

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4553 - 4569 December 7-11, 2022 ©2022 Association for Computational Linguistics Is a Question Decomposition Unit All We Need? Pruthvi Patel∗ Swaroop Mishra∗ Mihir Parmar Chitta Baral Arizona State University Abstract Large Language Models (LMs) have achieved state-of-the-art performance on many Natural Language Processing (NLP) benchmarks. With the growing number of new benchmarks, we build bigger and more complex LMs. How- ever, building new LMs may not be an ideal option owing to the cost, time and environmen- tal impact associated with it. We explore an alternative route: can we modify data by ex- pressing it in terms of the model’s strengths, so that a question becomes easier for models to answer? We investigate if humans can de- compose a hard question into a set of simpler questions that are relatively easier for models to solve. We analyze a range of datasets involv- ing various forms of reasoning and find that it is indeed possible to significantly improve model performance (24% for GPT3 and 29% for RoBERTa-SQuAD along with a symbolic calculator) via decomposition. Our approach provides a viable option to involve people in NLP research in a meaningful way. Our find- ings indicate that Human-in-the-loop Question Decomposition (HQD) can potentially provide an alternate path to building large LMs1. 1 Introduction With the advent of large LMs, we have achieved state-of-the-art performance on many NLP bench- marks (Radford et al., 2019; Brown et al., 2020; Sanh et al., 2021a). Our benchmarks are evolv- ing and becoming harder over time. To solve new benchmarks, we have been designing more com- plex and bigger LMs at the cost of computational resources, time and its negative impact on the en- vironment. Building newer LMs for solving new benchmarks may not be an ideal and sustainable option over time. Inspired by humans, who often view new tasks as a combination of existing tasks, we explore if we can mimic humans and help the ∗Equal Contribution 1https://github.com/Pruthvi98/QuestionDecomposition model solve a new task by decomposing (Mishra et al., 2021a) it as a combination of tasks that the model excels at and already knows. As NLP applications are increasingly more and more popular among people in their daily activi- ties, it is essential to develop methods that involve humans in NLP-powered applications in meaning- ful ways. Our approach attempts to fill this gap in LMs by providing a human-centric approach to modifying data. Solving complex QA tasks such as multi-hop QA, and numerical reasoning has been a challenge for models. Question Decomposition (QD) has recently been explored to empower mod- els to solve these tasks with the added advantage of interpretability. However, previous studies on QD are limited to some specific datasets (Khot et al., 2020b) such as DROP (Dua et al., 2019) and HOT- POTQA (Yang et al., 2018). We analyze a range of datasets involving various forms of reasoning to investigate if “a Question Decomposition Unit All We Need?" Figure 1 shows …

Content truncated to 500 words