Visually Grounded Neural Syntax Acquisition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1842–1861 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 1842 Visually Grounded Neural Syntax Acquisition Haoyue Shi†,∗ Jiayuan Mao‡,∗ Kevin Gimpel† Karen Livescu† †: Toyota Technological Institute at Chicago, IL, USA ‡: ITCS, Institute for Interdisciplinary Information Sciences, Tsinghua University, China {freda, kgimpel, klivescu}@ttic.edu, mjy14@mails.tsinghua.edu.cn Abstract We present the Visually Grounded Neural Syntax Learner (VG-NSL), an approach for learning syntactic representations and struc- tures without explicit supervision. The model learns by looking at natural images and read- ing paired captions. VG-NSL generates con- stituency parse trees of texts, recursively com- poses representations for constituents, and matches them with images. We define the concreteness of constituents by their matching scores with images, and use it to guide the parsing of text. Experiments on the MSCOCO data set show that VG-NSL outperforms var- ious unsupervised parsing approaches that do not use visual grounding, in terms of F1 scores against gold parse trees. We find that VG- NSL is much more stable with respect to the choice of random initialization and the amount of training data. We also find that the con- creteness acquired by VG-NSL correlates well with a similar measure defined by linguists. Fi- nally, we also apply VG-NSL to multiple lan- guages in the Multi30K data set, showing that our model consistently outperforms prior un- supervised approaches.1 1 Introduction We study the problem of visually grounded syn- tax acquisition. Consider the images in Figure 1, paired with the descriptive texts (captions) in En- glish. Given no prior knowledge of English, and sufficient such pairs, one can infer the correspon- dence between certain words and visual attributes, (e.g., recognizing that “a cat” refers to the objects in the blue boxes). One can further extract con- stituents, by assuming that concrete spans of words should be processed as a whole, and thus form the ∗HS and JM contributed equally to the work. 1 Project page: https://ttic.uchicago.edu/ ˜freda/project/vgnsl A cat stands under an umbrella. A cat is on the ground. A dog sits under an umbrella. Figure 1: We propose to use image-caption pairs to extract constituents from text, based on the assumption that similar spans should be matched to similar visual objects and these concrete spans form constituents. constituents. Similarly, the same process can be applied to verb or prepositional phrases. This intuition motivates the use of image-text pairs to facilitate automated language learning, in- cluding both syntax and semantics. In this paper we focus on learning syntactic structures, and pro- pose the Visually Grounded Neural Syntax Learner (VG-NSL, shown in Figure 2). VG-NSL acquires syntax, in the form of constituency parsing, by looking at images and reading captions. At a high level, VG-NSL builds latent con- stituency trees of word sequences and recursively composes representations for constituents. Next, it matches the visual and textual representations. The training procedure is built on the hypothesis that a better syntactic structure contributes to a better representation …

Content truncated to 500 words