Dense Procedure Captioning in Narrated Instructional Videos

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6382–6391 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6382 Dense Procedure Captioning in Narrated Instructional Videos Botian Shi1∗†, Lei Ji2,3†, Yaobo Liang3, Nan Duan3, Peng Chen4, Zhendong Niu1‡, Ming Zhou3 1Beijing Institute of Technology, Beijing, China 2Institute of Computing Technology, Chinese Academy of Science, Beijing, China 3Microsoft Research Asia, Beijing, China 4Microsoft Research and AI Group, Beijing, China botianshi@bit.edu.cn,{leiji,yalia,nanduan,peche}@microsoft.com zniu@bit.edu.cn,mingzhou@microsoft.com Abstract Understanding narrated instructional videos is important for both research and real-world web applications. Motivated by video dense captioning, we propose a model to gener- ate procedure captions from narrated instruc- tional videos which are a sequence of step- wise clips with description. Previous works on video dense captioning learn video seg- ments and generate captions without consid- ering transcripts. We argue that transcripts in narrated instructional videos can enhance video representation by providing fine-grained complimentary and semantic textual informa- tion. In this paper, we introduce a framework to (1) extract procedures by a cross-modality module, which fuses video content with the entire transcript; and (2) generate captions by encoding video frames as well as a snippet of transcripts within each extracted procedure. Experiments show that our model can achieve state-of-the-art performance in procedure ex- traction and captioning, and the ablation stud- ies demonstrate that both the video frames and the transcripts are important for the task. 1 Introduction Narrated instructional videos provide rich visual, acoustic and language information for people to easily understand how to complete a task by pro- cedures. An increasing amount of people resort to narrated instructional videos to learn skills and solve problems. For example, people would like to watch videos to repair a water damaged plas- terboard / drywall ceiling1 or cook Cottage Pie2. This motivates us to investigate whether machines can understand narrated instructional videos like ∗This work was done during the first author’s intership in MSR Asia †Equal contribution ‡Corresponding Author 1https://goo.gl/QZFsfR 2https://goo.gl/2Z4Kb8 Video Clip Time Transcript ... get a little pecorino Romano then use three egg yolks ... use yolk and whip these eggs up together ... now fix your spaghetti and boil water … put sauce on top pasta ... grate some pecorino cheese and beat the eggs stir cheese into the eggs cook the spaghetti in the boiling water pour the egg sauce on the spaghetti and mix well ... ... ... ... Procedure and Captions 0:00:12 0:00:46 0:00:52 0:01:10 0:01:27 0:02:04 0:02:16 0:02:30 Figure 1: A showcase of video dense procedure cap- tioning. In this task, the video frames and the transcript are given to (1) extract procedures in the video, (2) gen- erate a descriptive and informative sentence as the cap- tion of each procedure. humans. Besides, watching a long video is time- consuming, captions of videos provide a quick overview of video content for people to learn the main steps rapidly. Inspired by this, our task is to generate procedure captions from narrated instruc- tional videos which …

Content truncated to 500 words