What is Learned in Visually Grounded Neural Syntax Acquisition

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2615–2635 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2615 What is Learned in Visually Grounded Neural Syntax Acquisition Noriyuki Kojima, Hadar Averbuch-Elor, Alexander Rush and Yoav Artzi Department of Computer Science and Cornell Tech, Cornell University {nk654,he93,arush}@cornell.edu {yoav}@cs.cornell.edu Abstract Visual features are a promising signal for learn- ing bootstrap textual models. However, black- box learning models make it difficult to iso- late the specific contribution of visual compo- nents. In this analysis, we consider the case study of the Visually Grounded Neural Syntax Learner (Shi et al., 2019), a recent approach for learning syntax from a visual training sig- nal. By constructing simplified versions of the model, we isolate the core factors that yield the model’s strong performance. Contrary to what the model might be capable of learning, we find significantly less expressive versions produce similar predictions and perform just as well, or even better. We also find that a sim- ple lexical signal of noun concreteness plays the main role in the model’s predictions as op- posed to more complex syntactic reasoning. 1 Introduction Language analysis within visual contexts has been studied extensively, including for instruction fol- lowing (e.g., Anderson et al., 2018b; Misra et al., 2017, 2018; Blukis et al., 2018, 2019), visual ques- tion answering (e.g., Fukui et al., 2016; Hu et al., 2017; Anderson et al., 2018a), and referring ex- pression resolution (e.g., Mao et al., 2016; Yu et al., 2016; Wang et al., 2016). While significant progress has been made on such tasks, the combi- nation of vision and language makes it particularly difficult to identify what information is extracted from the visual context and how it contributes to the language understanding problem. Recently, Shi et al. (2019) proposed using align- ments between phrases and images as a learning signal for syntax acquisition. This task has been long-studied from a text-only setting, including re- cently using deep learning based approaches (Shen et al., 2018a, 2019; Kim et al., 2019; Havrylov et al., 2019; Drozdov et al., 2019, inter alia). While the introduction of images provides a rich new signal for the task, it also introduces numerous challenges, such as identifying objects and analyzing scenes. In this paper, we analyze the Visually Grounded Neural Syntax Learner (VG-NSL) model of Shi et al. (2019). In contrast to the tasks commonly studied in the intersection of vision and language, the existence of an underlying syntactic formalism allows for careful study of the contribution of the vi- sual signal. We identify the key components of the model and design several alternatives to reduce the expressivity of the model, at times, even replacing them with simple non-parameterized rules. This allows us to create several model variants, compare them with the full VG-NSL model, and visualize the information captured by the model parameters. Broadly, while we would expect a parsing model to distinguish between tokens and phrases along multiple dimensions to represent different syntactic roles, …

Content truncated to 500 words