Multimodal Event Transformer for Image-guided Story Ending Generation

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3434–3444 May 2-6, 2023 ©2023 Association for Computational Linguistics Multimodal Event Transformer for Image-guided Story Ending Generation Yucheng Zhou, Guodong Long∗ Australian AI Institute, School of Computer Science, FEIT, University of Technology Sydney yucheng.zhou-1@student.uts.edu.au, guodong.long@uts.edu.au Abstract Image-guided story ending generation (IgSEG) is to generate a story ending based on given story plots and ending image. Existing methods focus on cross-modal feature fusion but over- look reasoning and mining implicit information from story plots and ending image. To tackle this drawback, we propose a multimodal event transformer, an event-based reasoning frame- work for IgSEG. Specifically, we construct vi- sual and semantic event graphs from story plots and ending image, and leverage event-based reasoning to reason and mine implicit infor- mation in a single modality. Next, we con- nect visual and semantic event graphs and uti- lize cross-modal fusion to integrate different- modality features. In addition, we propose a multimodal injector to adaptive pass essential information to decoder. Besides, we present an incoherence detection to enhance the un- derstanding context of a story plot and the robustness of graph modeling for our model. Experimental results show that our method achieves state-of-the-art performance for the image-guided story ending generation. 1 Introduction Story ending generation (Guan et al., 2019) aims to generate a reasonable ending for a given story plot. It requires deep models to integrate powerful language understanding capability, which is crucial for artificial intelligence. Many efforts (Wang and Wan, 2019; Guan et al., 2019; Yao et al., 2019; Guan et al., 2020) have been proposed and achieved promising results since neural models designed for comprehending natural language allow them to understand story plots and reason reasonable story endings. With the advance of automatic story generation, it has attracted outstanding attention in multimodality research (Jung et al., 2020; Yu et al., 2021; Chen et al., 2021). ∗Corresponding author. It was our first big backyard barbeque of summer and we invited all friends. We all sat around and caught up with each others’ lives. Dave started the fire pit, look at those flames! Everyone put hot dogs on skewers and roasted them over the fire. We all had a great time hanging out until very late in the night and it was a great party! Story Plot Ending Image Story Ending Generation Figure 1: Given a multi-sentence story plot and an end- ing image, the image-guided story ending generation aims to generate a story ending related to the image. However, since story plots and story ending usu- ally correspond to different content, the context with information bottleneck is not enough to de- duce an informative story ending, i.e., generated endings tend to be inane and generic. To address this issue, Huang et al. (2021) propose an image- guided story ending generation (IgSEG) task that combines story plots and ending image to gener- ate a coherent, specific and informative story end- ing. IgSEG demands not only introducing informa- tion …

Content truncated to 500 words