Video-Grounded Dialogues with Pretrained Generation Language Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5842–5848 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5842 Video-Grounded Dialogues with Pretrained Generation Language Models Hung Le†∗, Steven C.H. Hoi†‡ †Singapore Management University hungle.2018@phdcs.smu.edu.sg ‡Salesforce Research Asia shoi@salesforce.com Abstract Pre-trained language models have shown re- markable success in improving various down- stream NLP tasks due to their ability to cap- ture dependencies in textual data and gener- ate natural responses. In this paper, we lever- age the power of pre-trained language mod- els for improving video-grounded dialogue, which is very challenging and involves com- plex features of different dynamics: (1) Video features which can extend across both spatial and temporal dimensions; and (2) Dialogue features which involve semantic dependencies over multiple dialogue turns. We propose a framework by extending GPT-2 models to tackle these challenges by formulating video- grounded dialogue tasks as a sequence-to- sequence task, combining both visual and tex- tual representation into a structured sequence, and fine-tuning a large pre-trained GPT-2 net- work. Our framework allows fine-tuning lan- guage models to capture dependencies across multiple modalities over different levels of in- formation: spatio-temporal level in video and token-sentence level in dialogue context. We achieve promising improvement on the Audio- Visual Scene-Aware Dialogues (AVSD) bench- mark from DSTC7, which supports a potential direction in this line of research. 1 Introduction Recent work in large-scale pre-training transformer- based neural networks (Liu et al., 2019; Devlin et al., 2019; Radford et al., 2019) has boosted the performance in various NLP tasks. The transformer-based architecture of these models al- lows them to capture various dependencies when trained on very large datasets. The pre-trained mod- els are adapted into downstream tasks to generate text that is more natural, fluent, and richer than ∗This work was mostly done when Hung Le was an intern at Salesforce Research Asia, Singapore. models not initialized with pre-trained weights. Similar to pre-trained CNN-based neural networks developed in computer vision research (He et al., 2016; Huang et al., 2017) which can learn high- resolution features in images, pre-trained language models (LMs) are capable of capturing fine-grain textual dependencies in text data of rich semantics. While the benefits of pre-trained language mod- els present in many downstream NLP tasks such as machine translation and question answering (QA) (Devlin et al., 2019; Lan et al., 2020), they are par- ticularly suitable to adapt to dialogue response gen- eration tasks for two major reasons: (1) Dialogue response generation usually involves more complex dynamics between input and output text sequences. The input typically involves dialogue history, in- cluding conversational exchanges between users and dialogue agents. A dialogue agent needs to capture relevant dependencies along each dialogue turns to generate a sensible response. (2) Compared to other NLP tasks, it is very challenging to collect and create large-scale dialogue datasets. Adopting pre-training approaches could ameliorate the lim- ited dialogue datasets by leveraging rich linguistic dependencies learned from other available text data. We are motivated by these observations …

Content truncated to 500 words