Hierarchical3D Adapters for Long Video-to-text Summarization

Findings of the Association for Computational Linguistics: EACL 2023, pages 1297–1320 May 2-6, 2023 ©2023 Association for Computational Linguistics Hierarchical3D Adapters for Long Video-to-text Summarization Pinelopi Papalampidi∗ Mirella Lapata Institute for Language, Cognition and Computation School of Informatics, University of Edinburgh p.papalampidi@sms.ed.ac.uk, mlap@inf.ed.ac.uk Abstract In this paper, we focus on video-to-text sum- marization and investigate how to best utilize multimodal information for summarizing long inputs (e.g., an hour-long TV show) into long outputs (e.g., a multi-sentence summary). We extend SummScreen (Chen et al., 2022), a dia- logue summarization dataset consisting of tran- scripts of TV episodes with reference sum- maries, and create a multimodal variant by col- lecting corresponding full-length videos. We incorporate multimodal information into a pre- trained textual summarizer efficiently using adapter modules augmented with a hierarchi- cal structure while tuning only 3.8% of model parameters. Our experiments demonstrate that multimodal adapters outperform more memory- heavy and fully fine-tuned textual summariza- tion methods. 1 Introduction What happens in the very last episode of “Friends”? Anyone who has seen this episode can summa- rize its key moments: Ross confesses his love for Rachel, they decide to resume their relationship, while Monica and Chandler adopt twins and move to the suburbs. TV viewers can naturally perform this dialogue summarization task having access to multiple modalities: they not only hear the actors speak but also see their expressions, actions, and whereabouts on screen. Despite recent advances in summarization (Nal- lapati et al., 2016; See et al., 2017; Liu and Lapata, 2019b) and increasing interest in different types of dialogue summarization, e.g., meeting transcripts (Gliwa et al., 2019; Zhong et al., 2021) or screen- plays (Chen et al., 2022), the contribution of modal- ities other than text remains relatively understudied. This is not entirely surprising given the challenges associated with the multimodal summarization task *Now at DeepMind. illustrated above (e.g., produce a written summary of a TV episode). Firstly, the input is long, it can- not fit into standard sequence-to-sequence archi- tectures, and the different modalities have to be somehow combined; secondly, the output is also long, summaries consist of multiple sentences and rich vocabulary; and thirdly, it involves complex inference over long-range dependencies between events and characters and common sense reasoning. At the same time, creating large-scale multimodal datasets with long videos and aligned textual data is challenging and time consuming, limiting the research conducted in this domain. Previous work on video-to-video summariza- tion identifies highlights from YouTube videos, TV shows, or movies (Song et al., 2015; Gygli et al., 2014; De Avila et al., 2011; Papalampidi et al., 2021b). However, in most cases, either the videos are short or the datasets are small with a few hun- dred examples. There is also limited work on video- to-text summarization. We are only aware of one large-scale multimodal dataset for this task, namely How2 (Sanabria et al., 2018), which again contains short videos (i.e., 2–3 minutes long) with simple semantics, and short, single-sentence summaries. In this paper, we focus on video-to-text summa- …

Content truncated to 500 words