Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 94–105 June 6–11, 2021. ©2021 Association for Computational Linguistics 94 Automatic Generation of Contrast Sets from Scene Graphs: Probing the Compositional Consistency of GQA Yonatan Bitton♦ Gabriel Stanovsky♦ Roy Schwartz♦ Michael Elhadad♠ ♦School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem, Israel ♠ Department of Computer Science, Ben Gurion University, Beer Sheva, Israel {yonatanbitton,gabis,roys}@cs.huji.ac.il elhadad@cs.bgu.ac.il Abstract Recent works have shown that supervised models often exploit data artifacts to achieve good test scores while their performance severely degrades on samples outside their training distribution. Contrast sets (Gardner et al., 2020) quantify this phenomenon by per- turbing test samples in a minimal way such that the output label is modified. While most contrast sets were created manually, requir- ing intensive annotation effort, we present a novel method which leverages rich semantic input representation to automatically gener- ate contrast sets for the visual question an- swering task. Our method computes the an- swer of perturbed questions, thus vastly re- ducing annotation cost and enabling thorough evaluation of models’ performance on vari- ous semantic aspects (e.g., spatial or rela- tional reasoning). We demonstrate the ef- fectiveness of our approach on the popular GQA dataset (Hudson and Manning, 2019) and its semantic scene graph image represen- tation. We find that, despite GQA’s composi- tionality and carefully balanced label distribu- tion, two strong models drop 13–17% in accu- racy on our automatically-constructed contrast set compared to the original validation set. Fi- nally, we show that our method can be applied to the training set to mitigate the degradation in performance, opening the door to more ro- bust models.1 1 Introduction NLP benchmarks typically evaluate in-distribution generalization, where test sets are drawn i.i.d from a distribution similar to the training set. Recent works showed that high performance on test sets sampled in this manner is often achieved by ex- ploiting systematic gaps, annotation artifacts, lex- ical cues and other heuristics, rather than learn- ing meaningful task-related signal. As a result, 1Our contrast sets and code are available at https://github.com/yonatanbitton/ AutoGenOfContrastSetsFromSceneGraphs. Figure 1: Illustration of our approach based on an ex- ample from the GQA dataset. Top: QA pairs and an image annotated with bounding boxes from the scene graph. Bottom: relations among the objects in the scene graph. First line at the top is the original QA pair, while the following 3 lines show our pertubated questions: replacing a single element in the question (a fence) with other options (a wall, men, an elephant), leading to a change in the output label. For each QA pair, the LXMERT predicted output is shown. the out-of-domain performance of these models is often severely deteriorated (Jia and Liang, 2017; Ribeiro et al., 2018; Gururangan et al., 2018; Geva et al., 2019; McCoy et al., 2019; Feng et al., 2019; Stanovsky et al., 2019). Recently, Kaushik et al. (2019) and Gardner et al. (2020) introduced the contrast sets …

Content truncated to 500 words