A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 419–429 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 419 A Case Study on Combining ASR and Visual Features for Generating Instructional Video Captions Jack Hessel Bo Pang Zhenhai Zhu Radu Soricut Cornell University Google jhessel@cs.cornell.edu {bopang,zhenhai,rsoricut}@google.com Abstract Instructional videos get high-traffic on video sharing platforms, and prior work suggests that providing time-stamped, subtask annota- tions (e.g., “heat the oil in the pan”) improves user experiences. However, current automatic annotation methods based on visual features alone perform only slightly better than con- stant prediction. Taking cues from prior work, we show that we can improve performance significantly by considering automatic speech recognition (ASR) tokens as input. Further- more, jointly modeling ASR tokens and visual features results in higher performance com- pared to training individually on either modal- ity. We find that unstated background infor- mation is better explained by visual features, whereas fine-grained distinctions (e.g., “add oil” vs. “add olive oil”) are disambiguated more easily via ASR tokens. 1 Introduction Instructional videos increasingly dominate user at- tention on online video platforms. For example, 86% of YouTube users report using the platform often to learn new things, and 70% of users report using videos to solve problems related to work, school, or hobbies (O’Neil-Hart, 2018). Prior work in user experience has investigated the best way of presenting instructional videos to users. Kim et al. (2014), for example, compare two options; first: presenting users with the video alone, and second: presenting the video with an additional structured representation, including a timeline populated with task subgoals. Users in- teracting with the structured video representation reported higher satisfaction, and external judges rated the work they completed using the videos as having higher quality. Margulieux et al. (2012) and Weir et al. (2015) similarly find that present- ing explicit subgoals alongside how-to videos im- ...knob of ginger and cut off a little bit and then just zest it... Cut up ginger and grate into the bowl Input: Target: ...best quality olive oil I can find... Heat some olive oil in a sauce pan Input: Target: ... that's perfection in my book right there, that's... Put the dish on a plate and serve Input: Target: Figure 1: Illustration of a multimodal dense instruc- tional video captioning task. Models are given access to both video frames and ASR tokens, and must gen- erate a recipe instruction step for each video segment. The speaker in the video sometimes (but not always) references literal objects and actions. proves user experiences. Thus, presenting instruc- tional videos with additional structured annota- tions is likely to benefit users. These studies rely on human annotation of time- stamped subtask goals, e.g., timed captions cre- ated through crowdsourcing. However, human- in-the-loop annotation is infeasible to deploy for popular video sharing platforms like YouTube that receive hundreds of hours of uploads per minute. In this work, we address the task of automatically producing captions for instructional videos at the level of …

Content truncated to 500 words