Psycholinguistics Meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3601–3605 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3601 Psycholinguistics meets Continual Learning: Measuring Catastrophic Forgetting in Visual Question Answering Claudio Greco1 claudio.greco@unitn.it Barbara Plank2 bplank@itu.dk Raquel Fernández3 raquel.fernandez@uva.nl Raffaella Bernardi1,4 raffaella.bernardi@unitn.it 1CIMeC and 4DISI University of Trento 2Dept. of Computer Science IT University of Copenhagen 3ILLC University of Amsterdam Abstract We study the issue of catastrophic forgetting in the context of neural multimodal approaches to Visual Question Answering (VQA). Moti- vated by evidence from psycholinguistics, we devise a set of linguistically-informed VQA tasks, which differ by the types of questions involved (Wh-questions and polar questions). We test what impact task difficulty has on con- tinual learning, and whether the order in which a child acquires question types facilitates com- putational models. Our results show that dra- matic forgetting is at play and that task diffi- culty and order matter. Two well-known cur- rent continual learning methods mitigate the problem only to a limiting degree. 1 Introduction Supervised machine learning models are incapable of continuously learning new tasks, as they for- get how to perform the previously learned ones. This problem, called catastrophic forgetting, is prominent in artificial neural networks (McClel- land et al., 1995). Continual Learning (CL) ad- dresses this problem by trying to equip models with the capability to continuously learn new tasks over time (Ring, 1997). Catastrophic forgetting and CL have received considerable attention in computer vision (e.g., Zenke et al., 2017; Kirk- patrick et al., 2017), but far less attention within Natural Language Processing (NLP). We investigate catastrophic forgetting in the context of multimodal models for Visual Question Answering (Antol et al., 2015) motivated by ev- idence from psycholinguistics. VQA is the task of answering natural language questions about an image. Evidence from child language acquisi- tion indicates that children learn Wh-questions before polar (Yes/No) questions (Moradlou and Ginzburg, 2016; Moradlou et al., 2018). Mo- tivated by this finding, we design a set of fA fB fCL .. .. .. .. Wh-q: y ∈ {metal, blue, sphere,..,large} Q: What is the material of the large object that is the same shape as the tiny yellow thing? A: metal Yes/No-q: y ∈ {Yes, No} Q: Does the cyan ball have the same material as the large object behind the green ball? A: Yes CLEVR (Johnson et al., 2017) Continual Learning (CL) - Single-head Setup: no task identifier provided tasks single softmax over all y Training phase Testing phase Multimodal tasks: Figure 1: Overview of our linguistically-informed CL setup for VQA. linguistically-informed experiments: i) to inves- tigate whether the order in which children ac- quire question types facilitates continual learning for computational models and, accordingly, the impact of task order on catastrophic forgetting; ii) to measure how far two well-known CL ap- proaches help to overcome the problem (Robins, 1995; Kirkpatrick et al., 2017)1. Contributions: Our study contributes to the lit- erature on CL in NLP. In particular: i) …

Content truncated to 500 words