Paired Examples as Indirect Supervision in Latent Decision Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5774–5785 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5774 Paired Examples as Indirect Supervision in Latent Decision Models Nitish Gupta1∗ Sameer Singh2 Matt Gardner3 Dan Roth1 1University of Pennsylvania, 2University of California, Irvine, 3Allen Institute for AI {nitishg, danroth}@seas.upenn.edu, sameer@uci.edu, mattg@allenai.org Abstract Compositional, structured models are appeal- ing because they explicitly decompose prob- lems and provide interpretable intermediate outputs that give confidence that the model is not simply latching onto data artifacts. Learn- ing these models is challenging, however, be- cause end-task supervision only provides a weak indirect signal on what values the latent decisions should take. This often results in the model failing to learn to perform the in- termediate tasks correctly. In this work, we introduce a way to leverage paired examples that provide stronger cues for learning latent decisions. When two related training exam- ples share internal substructure, we add an ad- ditional training objective to encourage con- sistency between their latent decisions. Such an objective does not require external super- vision for the values of the latent output, or even the end task, yet provides an additional training signal to that provided by individ- ual training examples themselves. We apply our method to improve compositional question answering using neural module networks on the DROP dataset. We explore three ways to acquire paired questions in DROP: (a) dis- covering naturally occurring paired examples within the dataset, (b) constructing paired ex- amples using templates, and (c) generating paired examples using a question generation model. We empirically demonstrate that our proposed approach improves both in- and out- of-distribution generalization and leads to cor- rect latent decision predictions. 1 Introduction Developing models that are capable of reasoning about complex real-world problems is challeng- ing. It involves decomposing the problem into sub- tasks, making intermediate decisions, and combin- ing them to make the final prediction. While many ∗ The first author is now affiliated with Google AI (guptanitish@google.com). approaches develop black-box models to solve such problems, we focus on compositional structured models as they provide a level of explanation for their predictions via interpretable latent decisions, and should, at least in theory, generalize better in compositional reasoning scenarios. For example, to answer How many field goals were scored in the first half? against a passage containing a football- game summary, a neural module network (NMN; Andreas et al., 2016) would first ground the set of field goals mentioned in the passage, then filter this set to the ones scored in the first half, and then return the size of the resulting set as the answer. Learning such models using just the end-task supervision is difficult, since the decision boundary that the model is trying to learn is complex, and the lack of any supervision for the latent decisions provides only a weak training signal. Moreover, the presence of dataset artifacts (Lai and Hockenmaier, 2014; Gururangan et al., 2018, among others), and degeneracy in the model, where incorrect …

Content truncated to 500 words