Visually Grounded Continual Learning of Compositional Phrases

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2018–2029, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 2018 Visually Grounded Continual Learning of Compositional Phrases Xisen Jin Junyi Du Arka Sadhu Ram Nevatia Xiang Ren Department of Computer Science, University of Southern California {xisenjin, junyidu, asadhu, nevatia, xiangren}@usc.edu Abstract Humans acquire language continually with much more limited access to data samples at a time, as compared to contemporary NLP sys- tems. To study this human-like language ac- quisition ability, we present VisCOLL, a visu- ally grounded language learning task, which simulates the continual acquisition of compo- sitional phrases from streaming visual scenes. In the task, models are trained on a paired image-caption stream which has shifting ob- ject distribution; while being constantly evalu- ated by a visually-grounded masked language prediction task on held-out test sets. VisCOLL compounds the challenges of continual learn- ing (i.e., learning from continuously shifting data distribution) and compositional general- ization (i.e., generalizing to novel composi- tions). To facilitate research on VisCOLL, we construct two datasets, COCO-shift and Flickr- shift, and benchmark them using different con- tinual learning methods. Results reveal that SoTA continual learning approaches provide little to no improvements on VisCOLL, since storing examples of all possible compositions is infeasible. We conduct further ablations and analysis to guide future work 1. 1 Introduction Modern NLP systems, including ones that build on pre-trained language models (Devlin et al., 2019; Radford et al., 2019), excel on a wide variety of tasks. These systems rely on offline (batch) training and have drawn recent criticism due to their inabil- ity to adapt to new contexts (Linzen, 2020). In con- trast, humans acquire language from evolving envi- ronments, require a small memory footprint (Mc- Clelland et al., 1995), and can generalize their knowledge to newer tasks (Sprouse et al., 2013). It has been suggested that humans ground perceptual 1Code and data: https://github.com/INK-USC/ VisCOLL A green with a bunch of MASK MASK hanging from it. green apple (b) Predicting Novel Composition VisCOLL Model VisCOLL Model Seen Compositions Seen Atomic Words green apple red train A hand holding a half eaten MASK MASK. red apple VisCOLL Model (a) Predicting Seen Composition Training Data Stream ... green train red apples Some fresh MASK MASK on the grass. A MASK MASK traveling down ... Seen Concepts green train red apples Figure 1: Illustration of the proposed VisCOLLtask. The end-task is Masked-Token Prediction: given an im- age, a model predicts the masked tokens of an associ- ated caption in an online continual learning setup (cf. (a) in the figure). Additionally, we test composition generalization by evaluating on novel compositions (b) which are not encountered at train time. experience to semantically interpret symbols (Bisk et al., 2020; Harnad, 1990; Vigliocco et al., 2014). Model model the challenge, we propose Vis- COLL, a Visually-grounded ContinuaL Learning setup, to acquire compositional phrases from streaming visual-linguistic data. Models receive a stream of paired image-caption data which has a shifting object distribution. As the end …

Content truncated to 500 words