Video-aided Unsupervised Grammar Induction

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1513–1524 June 6–11, 2021. ©2021 Association for Computational Linguistics 1513 Video-aided Unsupervised Grammar Induction Songyang Zhang1∗, Linfeng Song2, Lifeng Jin2, Kun Xu2, Dong Yu2 and Jiebo Luo1 1University of Rochester, Rochester, NY, USA szhang83@ur.rochester.edu, jluo@cs.rochester.edu 2Tencent AI Lab, Bellevue, WA, USA {lfsong,lifengjin,kxkunxu,dyu}@tencent.com Abstract We investigate video-aided grammar induc- tion, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal gram- mar induction focus on learning syntactic grammars from text-image pairs, with promis- ing results showing that the information from static images is useful in induction. However, videos provide even richer information, includ- ing not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Com- pound PCFG model (Kim et al., 2019) as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to ef- fectively aggregate these rich features from dif- ferent modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each in- dividual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the ef- fectiveness of leveraging video information for unsupervised grammar induction. 1 Introduction Constituency parsing is an important task in nat- ural language processing, which aims to capture syntactic information in sentences in the form of constituency parsing trees. Many conventional ap- proaches learn constituency parser from human- annotated datasets such as Penn Treebank (Marcus et al., 1993). However, annotating syntactic trees by human language experts is expensive and time- consuming, while the supervised approaches are limited to several major languages. In addition, the treebanks for training these supervised parsers are small in size and restricted to the newswire domain, thus their performances tend to be worse ∗ This work was done when Songyang Zhang was an intern at Tencent AI Lab. (a) Sentence: Man starts to play the guitar fast. Sentence: A squirrel jumps on stump. (b) Bird sound Bird sound Guitar sound Guitar sound Figure 1: Examples of video aided unsupervised gram- mar induction. We aim to improve the constituency parser by leveraging aligned video-sentence pairs. when applying to other domains (Fried et al., 2019). To address these issues, recent approaches (Shen et al., 2018b; Jin et al., 2018; Drozdov et al., 2019; Kim et al., 2019) design unsupervised constituency parsers and grammar inducers, since they can be trained on large-scale unlabeled data. In partic- ular, there has been growing interests in exploit- ing visual information for unsupervised grammar induction because visual information can capture important knowledge required for language learn- ing that is ignored by text (Gleitman, 1990; Pinker and MacWhinney, 1987; Tomasello, 2003). This task aims to learn a constituency parser from raw unlabeled text aided by its visual context. Previous methods (Shi et al., 2019; Kojima et al., 2020; Zhao and Titov, 2020; …

Content truncated to 500 words