Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4664–4677, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4664 Generating Image Descriptions via Sequential Cross-Modal Alignment Guided by Human Gaze Ece Takmaz1, Sandro Pezzelle1, Lisa Beinborn2, Raquel Fern´andez1 1Institute for Logic, Language and Computation, University of Amsterdam 2Vrije Universiteit Amsterdam {e.takmaz|s.pezzelle|raquel.fernandez}@uva.nl, l.m.beinborn@vu.nl Abstract When speakers describe an image, they tend to look at objects before mentioning them. In this paper, we investigate such sequential cross- modal alignment by modelling the image de- scription generation process computationally. We take as our starting point a state-of-the- art image captioning system and develop sev- eral model variants that exploit information from human gaze patterns recorded during lan- guage production. In particular, we propose the first approach to image description gener- ation where visual processing is modelled se- quentially. Our experiments and analyses con- firm that better descriptions can be obtained by exploiting gaze-driven attention and shed light on human cognitive processes by comparing different ways of aligning the gaze modality with language production. We find that pro- cessing gaze data sequentially leads to descrip- tions that are better aligned to those produced by speakers, more diverse, and more natural— particularly when gaze is encoded with a dedi- cated recurrent component. 1 Introduction Describing an image requires the coordination of different modalities. There is a long tradition of cognitive studies showing that the interplay be- tween language and vision is complex. On the one hand, eye movements are influenced by the task at hand, such as locating objects or verbally describ- ing an image (Buswell, 1935; Yarbus, 1967). On the other hand, visual information processing plays a role in guiding linguistic production (e.g., Griffin, 2004; Gleitman et al., 2007). Such cross-modal co- ordination unfolds sequentially in the specific task of image description (Coco and Keller, 2012)—i.e., objects tend to be looked at before being mentioned. Yet, the temporal alignment between the two modal- ities is not straightforward (Griffin and Bock, 2000; Vaidyanathan et al., 2015) Figure 1: In our approach, an image captioning model is fed with a sequence of masked images encoding the gaze fixations of a single human speaker during lan- guage production. This diagram is a toy illustration. In this paper, we follow up on these findings and investigate cross-modal alignment in image de- scription by modelling the description generation process computationally. We take a state-of-the-art system for automatic image captioning (Anderson et al., 2018) and develop several model variants that exploit information derived from eye-tracking data. To train these models, we use a relatively small dataset of image descriptions in Dutch (DIDEC; van Miltenburg et al., 2018) that includes informa- tion on gaze patterns collected during language pro- duction. We hypothesise that a system that encodes gaze data as a proxy for human visual attention will lead to better, more human-like descriptions. In particular, we propose that training such a sys- tem with eye-movements sequentially aligned with utterances (see Figure 1) will produce descriptions that reflect …

Content truncated to 500 words