Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2223 - 2235 December 7-11, 2022 ©2022 Association for Computational Linguistics Learning to Decompose: Hypothetical Question Decomposition Based on Comparable Texts Ben Zhou1∗ Kyle Richardson2 Xiaodong Yu1 Dan Roth1 1University of Pennsylvania 2Allen Institute for AI {xyzhou, xdyu, danroth}@seas.upenn.edu kyler@allenai.org Abstract Explicit decomposition modeling, which in- volves breaking down complex tasks into more straightforward and often more interpretable sub-tasks, has long been a central theme in developing robust and interpretable NLU sys- tems. However, despite the many datasets and resources built as part of this effort, the ma- jority have small-scale annotations and lim- ited scope, which is insufficient to solve gen- eral decomposition tasks. In this paper, we look at large-scale intermediate pre-training of decomposition-based transformers using dis- tant supervision from comparable texts, par- ticularly large-scale parallel news. We show that with such intermediate pre-training, devel- oping robust decomposition-based models for a diverse range of tasks becomes more feasi- ble. For example, on semantic parsing, our model, DECOMPT5, improves 20% to 30% on two datasets, Overnight and TORQUE, over the baseline language model. We further use DECOMPT5 to build a novel decomposition- based QA system named DECOMPENTAIL, improving over state-of-the-art models, includ- ing GPT-3, on both HotpotQA and StrategyQA by 8% and 4%, respectively. 1 Introduction Answering questions often involves making edu- cated guesses: we do not necessarily have accurate facts but can use common sense to understand what most questions are asking and what kinds of knowl- edge are needed. For example (see Fig. 1), we can understand the question “Is Albany, GA more crowded than Albany, NY?” involves comparing the size and population of two cities without know- ing the specific numbers to compare. It is often desirable to make such a decomposition because a city’s population is usually much easier to acquire than a direct answer to the original question. Existing approaches to end-to-end question- answering (QA) assume that pre-trained language ∗Work partly done when interning at AI2. Is Albany, GA more crowded than Albany, NY? ... Tokyo is a crowded city. … …Tokyo is a small city that faces over-population issues… Doc Group A Doc Group B Parallel News Distantly-supervised Model for Query Understanding Related to the area size Related the number of people Better latent representation for decomposition Albany, GA has 72,000 people. Albany, GA is 55 mi2 in size… Answer: No Actual Question Figure 1: An example of how parallel news documents can be used to train a model that is capable of making educated guesses on what the question is asking, and how it may help to derive a better answer. models (LMs) are capable of both robust question understanding of this type and acquiring the rel- evant facts. Much recent evidence, however, has revealed limitations in the commonsense and com- positional reasoning abilities of current transform- ers (Zhou et al., 2019; Liu et al., 2021), in part due to reporting biases (e.g., relating the semantics of “more crowded” …

Content truncated to 500 words