MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2603–2614 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 2603 MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning Jie Lei1∗ Liwei Wang2 Yelong Shen3∗ Dong Yu2 Tamara L. Berg1 Mohit Bansal1 1UNC Chapel Hill 2Tencent AI Lab Seattle USA 3Microsoft Dynamics 365 AI {jielei, tlberg, mbansal}@cs.unc.edu {liweiwang, dyu}@tencent.com, {yeshe}@microsoft.com Abstract Generating multi-sentence descriptions for videos is one of the most challenging cap- tioning tasks due to its high requirements for not only visual relevance but also discourse- based coherence across the sentences in the paragraph. Towards this goal, we propose a new approach called Memory-Augmented Re- current Transformer (MART), which uses a memory module to augment the transformer architecture. The memory module generates a highly summarized memory state from the video segments and the sentence history so as to help better prediction of the next sentence (w.r.t. coreference and repetition aspects), thus encouraging coherent paragraph genera- tion. Extensive experiments, human evalua- tions, and qualitative analyses on two popular datasets ActivityNet Captions and YouCookII show that MART generates more coherent and less repetitive paragraph captions than base- line methods, while maintaining relevance to the input video events.1 1 Introduction In video captioning, the task is to generate a natu- ral language description capturing the content of a video. Recently, dense video captioning (Krishna et al., 2017) has emerged as an important task in this field, where systems first generate a list of tem- poral event segments from a video, then decode a coherent paragraph (multi-sentence) description from the generated segments. Park et al. (2019) simplifies this task as generating a coherent para- graph from a provided list of segments, removing the requirements for generating the event segments, and focusing on decoding better paragraph cap- tions from the segments. As noted by Xiong et al. ∗ Work done while Jie Lei was an intern and Yelong Shen was an employee at Tencent AI Lab. 1All code is available open-source at https://github. com/jayleicn/recurrent-transformer (2018); Park et al. (2019), generating paragraph descriptions for videos can be very challenging due to the difficulties of having relevant, less redundant, as well as coherent generated sentences. Towards this goal, Xiong et al. (2018) proposed a variant of the LSTM network (Hochreiter and Schmidhuber, 1997) that generates a new sentence conditioned on previously generated sentences by passing the LSTM hidden states throughout the entire decoding process. Park et al. (2019) further augmented the above LSTM caption generator with a set of three discriminators that score generated sentences based on defined metrics, i.e., relevance, linguistic diversity, and inter-sentence coherence. Though different, both these methods use LSTMs as the language decoder. Recently, transformers (Vaswani et al., 2017) have proven to be more effective than RNNs (e.g., LSTM (Hochreiter and Schmidhuber, 1997), GRU (Chung et al., 2014), etc.), demonstrating su- perior performance in many sequential modeling tasks (Vaswani et al., 2017; Zhou et al., 2018; De- vlin et al., 2019; Dai et al., 2019; Yang …

Content truncated to 500 words