IgSEG: Image-guided Story Ending Generation

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3114–3123 August 1–6, 2021. ©2021 Association for Computational Linguistics 3114 IgSEG: Image-guided Story Ending Generation Qingbao Huang1,2,6, Chuan Huang1, Linzhang Mo1 Jielong Wei1, Yi Cai2,5∗, Ho-fung Leung3, Qing Li4 1School of Electrical Engineering, Guangxi University, Nanning, Guangxi, China 2School of Software Engineering, South China University of Technology, Guangzhou, China 3Dept. of Computer Sc. & Engin, The Chinese University of Hong Kong, Hong Kong, China 4Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China 5Key Laboratory of Big Data and Intelligent Robot (SCUT), MOE of China 6Institute of Artificial Intelligence, Guangxi University, Nanning Guangxi, China qbhuang@gxu.edu.cn, chuanhuangqy@gmail.com, mlz1997@163.com Abstract In this work, we propose a new task call- ed Image-guided Story Ending Generation (IgSEG). Given a multi-sentence story plot and an ending-related image, IgSEG aims to gen- erate a story ending that conforms to the con- textual logic and the relevant visual concepts. In contrast to the story ending generation task, which generates open-ended endings, the ma- jor challenges of IgSEG are to comprehend the given context and image sufficiently, and mine the appropriate semantics from the image to make the generated story ending informative, reasonable, and coherent. To address the chal- lenges, we propose a Multi-layer Graph con- volution and Cascade-LSTM (MGCL) based model which mainly comprises of two collab- orative modules: i) a multi-layer graph convo- lutional network to learn the dependency rela- tions of sentences and the logical clue of the context; ii) a multiple context-image attention module to generate the story endings by gradu- ally incorporating textual and visual semantic concepts. Our MGCL is thus capable of build- ing logically consistent and semantically rich story endings. To evaluate the proposed model, we modify the existing VIST dataset to ob- tain the VIST-Ending dataset. Empirically, our MGCL outperforms all the strong baselines on both automatic and human evaluation. 1 Introduction As two challenging subtasks of story generation, the story ending generation (SEG) and visual story- telling (Huang et al., 2016; Zhao et al., 2018) have attracted more attention recently. The former gen- erates text-based story endings (Zhao et al., 2018; Li et al., 2018; Guan et al., 2019). While, the lat- ter generates photo-streams-based stories (Huang et al., 2016; Wang et al., 2018; Hu et al., 2020) or one-image-based stories (Gaur, 2019). Distinctly, ∗Corresponding author: Cai Yi (ycai@scut.edu.cn). I went to a gun training class. I was not the only one, my friends came with me. We inspected ammo. And we learn how to shoot at targets. (a) (b) (c) (d) (e) IC IC SEG SEG We were proud of our hits. We were proud of our hits. Two people hold- ing paper targets. Two people hold- ing paper targets. It was a happy day. It was a happy day. IgSEG IgSEG Figure 1: In SEG, existing methods tend to generate generic, safe, and inane story endings, e.g., (c). IgSEG is designed to generate specific, reasonable and infor- mative endings induced by the given ending-related im- …

Content truncated to 500 words