What’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision

Human Language Technologies: The 2015 Annual Conference of the North American Chapter of the ACL, pages 143–152, Denver, Colorado, May 31 – June 5, 2015. c⃝2015 Association for Computational Linguistics What’s Cookin’? Interpreting Cooking Videos using Text, Speech and Vision Jonathan Malmaud, Jonathan Huang, Vivek Rathod, Nick Johnston, Andrew Rabinovich, and Kevin Murphy Google 1600 Amphitheatre Parkway Mountain View, CA 94043 malmaud@mit.edu {jonathanhuang, rathodv, nickj, amrabino, kpmurphy}@google.com Abstract We present a novel method for aligning a se- quence of instructions to a video of some- one carrying out a task. In particular, we fo- cus on the cooking domain, where the instruc- tions correspond to the recipe. Our technique relies on an HMM to align the recipe steps to the (automatically generated) speech tran- script. We then refine this alignment using a state-of-the-art visual food detector, based on a deep convolutional neural network. We show that our technique outperforms simpler techniques based on keyword spotting. It also enables interesting applications, such as auto- matically illustrating recipes with keyframes, and searching within a video for events of in- terest. 1 Introduction In recent years, there have been many successful attempts to build large “knowledge bases” (KBs), such as NELL (Carlson et al., 2010), KnowItAll (Et- zioni et al., 2011), YAGO (Suchanek et al., 2007), and Google’s Knowledge Graph/ Vault (Dong et al., 2014). These KBs mostly focus on declarative facts, such as “Barack Obama was born in Hawaii”. But human knowledge also encompasses procedural in- formation not yet within the scope of such declara- tive KBs – instructions and demonstrations of how to dance the tango, for example, or how to change a tire on your car. A KB for organizing and retrieving such procedural knowledge could be a valuable resource for helping people (and potentially even robots – e.g., (Saxena et al., 2014; Yang et al., 2015)) learn to perform various tasks. In contrast to declarative information, procedural knowledge tends to be inherently multimodal. In particular, both language and perceptual information are typically used to parsimoniously describe proce- dures, as evidenced by the large number of “how- to” videos and illustrated guides on the open web. To automatically construct a multimodal database of procedural knowledge, we thus need tools for extracting information from both textual and vi- sual sources. Crucially, we also need to figure out how these various kinds of information, which often complement and overlap each other, fit together to a form a structured knowledge base of procedures. As a small step toward the broader goal of align- ing language and perception, we focus in this pa- per on the problem of aligning video depictions of procedures to steps in an accompanying text that corresponds to the procedure. We focus on the cooking domain due to the prevalence of cooking videos on the web and the relative ease of inter- preting their recipes as linear sequences of canon- ical actions. In this domain, the textual source is a user-uploaded recipe attached to the video show- ing the recipe’s …

Content truncated to 500 words