Grounded PCFG Induction with Images

Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing, pages 396–408 December 4 - 7, 2020. c⃝2020 Association for Computational Linguistics 396 Grounded PCFG Induction with Images Lifeng Jin and William Schuler Department of Linguistics The Ohio State University, Columbus, OH, USA {jin, schuler}@ling.osu.edu Abstract Recent work in unsupervised parsing has tried to incorporate visual information into learn- ing, but results suggest that these models need linguistic bias to compete against models that only rely on text. This work proposes gram- mar induction models which use visual infor- mation from images for labeled parsing, and achieve state-of-the-art results on grounded grammar induction on several languages. Re- sults indicate that visual information is espe- cially helpful in languages where high fre- quency words are more broadly distributed. Comparison between models with and without visual information shows that the grounded models are able to use visual information for proposing noun phrases, gathering useful in- formation from images for unknown words, and achieving better performance at preposi- tional phrase attachment prediction.1 1 Introduction Recent grammar induction models are able to pro- duce accurate grammars and labeled parses with raw text only (Jin et al., 2018b, 2019; Kim et al., 2019b,a; Drozdov et al., 2019), providing evi- dence against the poverty of the stimulus argument (Chomsky, 1965), and showing that many linguistic distinctions like lexical and phrasal categories can be directly induced from raw text statistics. How- ever, as computational-level models of human syn- tax acquisition, they lack semantic, pragmatic and environmental information which human learners seem to use (Gleitman, 1990; Pinker and MacWhin- ney, 1987; Tomasello, 2003). This paper proposes novel grounded neural- network-based models of grammar induction which take into account information extracted from im- ages in learning. Performance comparisons show 1The system implementation and translated datasets used in this work can be found at https://github.com/ lifengjin/imagepcfg. (a) friend as companion (b) friend as condiment Figure 1: Examples of disambiguating information pro- vided by images for the prepositional phrase attach- ment of the sentence Mary eats spaghetti with a friend (Gokcen et al., 2018). that the proposed models achieve state-of-the-art re- sults on multilingual induction datasets, even with- out help from linguistic knowledge or pretrained image encoders. Experiments show several specific benefits attributable to the use of visual informa- tion in induction. First, as a proxy to semantics, the co-occurrences between objects in images and referring words and expressions, such as the word spaghetti and the plate of spaghetti in Figure 1,2 provide clues to the induction model about the syn- tactic categories of such linguistic units, which may complement distributional cues from word collo- cation which normal grammar inducers rely on solely for induction. Also, pictures may help dis- ambiguate different syntactic relations: induction models are not able to resolve many prepositional phrase attachment ambiguities with only text — for example in Figure 1, there is little information in the text of Mary eats spaghetti …

Content truncated to 500 words