Visually Grounded Compound PCFGs

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4369–4379, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4369 Visually Grounded Compound PCFGs Yanpeng ZhaoE EILCC, University of Edinburgh yanp.zhao@ed.ac.uk Ivan TitovEæ æILLC, University of Amsterdam ititov@inf.ed.ac.uk Abstract Exploiting visual groundings for language un- derstanding has recently been drawing much attention. In this work, we study visually grounded grammar induction and learn a con- stituency parser from both unlabeled text and its visual groundings. Existing work on this task (Shi et al., 2019) optimizes a parser via REINFORCE and derives the learning signal only from the alignment of images and sen- tences. While their model is relatively accu- rate overall, its error distribution is very un- even, with low performance on certain con- stituents types (e.g., 26.2% recall on verb phrases, VPs) and high on others (e.g., 79.6% recall on noun phrases, NPs). This is not surprising as the learning signal is likely in- sufficient for deriving all aspects of phrase- structure syntax and gradient estimates are noisy. We show that using an extension of probabilistic context-free grammar model we can do fully-differentiable end-to-end visually grounded learning. Additionally, this enables us to complement the image-text alignment loss with a language modeling objective. On the MSCOCO test captions, our model estab- lishes a new state of the art, outperforming its non-grounded version and, thus, confirm- ing the effectiveness of visual groundings in constituency grammar induction. It also sub- stantially outperforms the previous grounded model, with largest improvements on more ‘abstract’ categories (e.g., +55.1% recall on VPs).1 1 Introduction Grammar induction is a task of finding latent hier- archical structure of language. As a fundamental problem in computational linguistics, it has been extensively studied for decades (Lari and Young, 1990; Carroll and Charniak, 1992; Clark, 2001; 1Our code is available at https://git.io/JU0JJ. Klein and Manning, 2002). Recently, deep learn- ing models have been shown very effective across NLP tasks and have also been applied to grammar induction, greatly advancing the area (Shen et al., 2018, 2019; Kim et al., 2019a,b; Jin et al., 2019). These neural grammar-induction approaches have been generally limited to relying on text, without considering learning signals from other modalities. In contrast, the crucial aspect of natural language learning is that it is grounded in perceptual ex- periences (Barsalou, 1999; Fincher-Kiefer, 2001; Bisk et al., 2020). We thus anticipate improved language understanding by leveraging grounded learning. Promising results from grounded learn- ing have been emerging in areas such as representa- tion learning (Bruni et al., 2014; Kiela et al., 2018; Bordes et al., 2019). Typically, they use visual im- ages as perceptual groundings of language and aim at improving continuous vector representations of language (e.g., word or sentence embeddings). In this work, we consider a more challenging prob- lem: can visual groundings help us induce syntac- tic structure? We refer to this problem as visually grounded grammar induction. Shi et al. (2019) propose a visually grounded neural syntax learner (VG-NSL) to tackle the task. Specifically, …

Content truncated to 500 words