Multimodal Persona Based Generation of Comic Dialogs

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 14150–14164 July 9-14, 2023 ©2023 Association for Computational Linguistics Multimodal Persona Based Generation of Comic Dialogs Harsh Agrawal IIT Delhi harsh.ag14901@gmail.com Aditya M. Mishra IIT Delhi mishramohanaditya@gmail.com Manish Gupta Microsoft gmanish@microsoft.com Mausam IIT Delhi mausam@cse.iitd.ac.in Abstract We focus on the novel problem of persona based dia- logue generation for comic strips. Dialogs in comic strips is a unique and unexplored area where every strip contains utterances from various characters with each one building upon the previous utterances and the associated visual scene. Previous works like Di- aloGPT, PersonaGPT and other dialog generation models encode two-party dialogues and do not ac- count for the visual information. To the best of our knowledge we are the first to propose the paradigm of multimodal persona based dialogue generation. We contribute a novel dataset, COMSET, consisting of 54K strips, harvested from 13 popular comics available online. Further, we propose a multimodal persona-based architecture, MPDIALOG, to gener- ate dialogues for the next panel in the strip which decreases the perplexity score by ∼10 points over strong dialogue generation baseline models. We demonstrate that there is still ample opportunity for improvement, highlighting the importance of build- ing stronger dialogue systems that are able to gener- ate persona-consistent dialogues and understand the context through various modalities. 1 Introduction Multimodal conversational agents build dialog sys- tems that engage with modalities beyond text, in constructing next responses. They open up a novel direction of text-vision multimodality, where the agent is part of the scene, rather than being a dis- tant observer. This facilitates research and creation of support based multimodal agents. These agents could be critical for various applications such as assistants for visually impaired, conversations with robots in physical settings, instruction following by a digital agent that is manipulating images, clar- ification discussions during a presentation and so on. Such agents can help to promote literacy and language skills, as users engage with the generated dialogue to create their own stories. In all such cases, a natural conversational experience will be Gold Response: It's a really convenient way for me to date other women... Persona Facts (Character: Tyr, Comic: Inkpen) • I am the Norse god of war, law and honor. • I am son of Odin, half-brother of Thor. • I was betrayed by my father. • I like to throw my weight around. • I am working on anger management. Comic Dialog Generation Model Panel Images Utterances Persona MPDialog (our work): She can see into space from her home planet I'll never get married again... ever.... oh well.. thanks for everything it really saved us some time... but we're gonna live together anyway... right??? Output Figure 1: Comic Dialogue Generation: Input is a comic strip, with its text utterances, segmented visual panels and persona of target character; output is an utterance. emulated better if visual or other modal elements get incorporated in the AI models. There is substantial recent …

Content truncated to 500 words