MPCHAT: Towards Multimodal Persona-Grounded Conversation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3354–3377 July 9-14, 2023 ©2023 Association for Computational Linguistics MPCHAT: Towards Multimodal Persona-Grounded Conversation Jaewoo Ahn1 Yeda Song1 Sangdoo Yun2,1 Gunhee Kim1 1Seoul National University 2NAVER AI Lab {jaewoo.ahn,yeda.song}@vision.snu.ac.kr, sangdoo.yun@navercorp.com, gunhee@snu.ac.kr http://vision.snu.ac.kr/projects/mpchat Abstract In order to build self-consistent personalized dialogue agents, previous research has mostly focused on textual persona that delivers per- sonal facts or personalities. However, to fully describe the multi-faceted nature of per- sona, image modality can help better reveal the speaker’s personal characteristics and ex- periences in episodic memory (Rubin et al., 2003; Conway, 2009). In this work, we extend persona-based dialogue to the multimodal do- main and make two main contributions. First, we present the first multimodal persona-based dialogue dataset named MPCHAT, which ex- tends persona with both text and images to contain episodic memories. Second, we em- pirically show that incorporating multimodal persona, as measured by three proposed mul- timodal persona-grounded dialogue tasks (i.e., next response prediction, grounding persona prediction, and speaker identification), leads to statistically significant performance improve- ments across all tasks. Thus, our work high- lights that multimodal persona is crucial for im- proving multimodal dialogue comprehension, and our MPCHAT serves as a high-quality re- source for this research. 1 Introduction With the rapid advance of conversational AI sys- tems in recent years, developing self-consistent di- alogue agents has been studied much (Li et al., 2016; Zhang et al., 2018). Considerable re- search aims to endow dialogue agents with per- sona, which represents an individual’s personality (Zhong et al., 2022; Cao et al., 2022). In partic- ular, researchers have exploited textual descrip- tion of persona, for example, in the form of un- structured sentences (Mazaré et al., 2018), struc- tured key-value attributes (e.g., age, gender, loca- tion) (Song et al., 2020) and personality types (e.g., Big-Five) (Mairesse and Walker, 2007). Therefore, dialogue agents with persona have been found to Persona image-sentence pairs (P) Dialogue example p1 p2 p3 p4 p5 image (pi) # sentence (pt) u/userB • 2 weeks ago u/userA • 2 weeks ago pic of a rocket launch from spaceX. i found this breathtaking. 42 comments Reply u/userB • 2 weeks ago Curious, what would you estimate the ratio of acceptable shots to unacceptable shots is? u/userA • 2 weeks ago cameras often take 100-200+ pictures by the noise of the vehicle. If one turns out acceptable, I wouldn't really call it a "1/200" keeper rate. i took a high dynamic range image of the solar eclipse, revealing lunar detail during totality. i photographed the milky way with a lighthouse in the foreground in sanibel island, florida tonight, i carved a pumpkin. user A 2 Reply 1 Reply 3 Reply hi! this is my photograph. feel free to see more of my work on my website : response is grounded on pm one of my recent favorites: long exposure of a falcon 9 rocket launch, reflecting in the water i placed a sound-activated camera …

Content truncated to 500 words