Structural and Functional Decomposition for Personality Image Captioning in a Communication Game

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4587–4593 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 4587 Structural and Functional Decomposition for Personality Image Captioning in a Communication Game Minh Thu Nguyen1, Duy Phung1, Minh Hoai2 and Thien Huu Nguyen3 1 VinAI Research, Vietnam 2 Stony Brook University, Stony Brook, New York, USA 3 Department of Computer and Information Science, University of Oregon, Eugene, OR 97403, USA {v.thunm15,v.duypv1}@vinai.io, minhhoai@cs.stonybrook.edu, thien@cs.uoregon.edu Abstract Personality image captioning (PIC) aims to de- scribe an image with a natural language cap- tion given a personality trait. In this work, we introduce a novel formulation for PIC based on a communication game between a speaker and a listener. The speaker attempts to gener- ate natural language captions while the listener encourages the generated captions to contain discriminative information about the input im- ages and personality traits. In this way, we expect that the generated captions can be im- proved to naturally represent the images and express the traits. In addition, we propose to adapt the language model GPT2 to perform caption generation for PIC. This enables the speaker and listener to benefit from the lan- guage encoding capacity of GPT2. Our exper- iments show that the proposed model achieves the state-of-the-art performance for PIC. 1 Introduction To effectively communicate with human, an im- portant step involves image captioning (IC) that requires systems to describe images using natu- ral language captions. Image captioning (IC) has been studied extensively, featuring deep learning models (i.e., the encoder-decoder architectures) as the dominant approach (Vinyals et al., 2014; Xu et al., 2015; Anderson et al., 2017; Yang et al., 2018). Despite its popularity, the current work on IC has mainly considered the factual setting for IC where the generated captions should faithfully present the visual content of images. A major lim- itation for this factual IC task concerns its failure to incorporate human factors (i.e., personalities or traits) into the caption generation process. As such, ones might prefer to produce engaging captions where his/her personality traits are explicitly ex- pressed and the visual concepts in the images are not necessarily covered in their full details. Con- sequently, in this work, we seek to fill in this gap for IC by exploring personality image captioning (PIC) where the models need to further consider a personality/trait in the captioning process. In par- ticular, we leverage PERSONALITY-CAPTIONS (PC) (Shuster et al., 2019), the first dataset for PIC, to evaluate the models in this work. Which characteristics should a caption have to adequately describe an image in PIC? Motivated by the functional and structural decomposition for lan- guage learning (Lazaridou et al., 2016, 2020; Kot- tur et al., 2017), we argue that an effective caption for PIC should posses two important properties. On the one hand, the captions in PIC should follow the natural language structures to induce effective com- munication with human (i.e., the structural view or naturalness of the captions). On the other hand, for the functional view, …

Content truncated to 500 words