Procedural Text Generation from an Execution Video

Proceedings of the The 8th International Joint Conference on Natural Language Processing, pages 326–335, Taipei, Taiwan, November 27 – December 1, 2017 c⃝2017 AFNLP Procedural Text Generation from an Execution Video Atsushi Ushiku Hayato Hashimoto Atsushi Hashimoto Shinsuke Mori Yoshida-honmachi, Sakyo-ku, Kyoto University, Kyoto, Japan d460655046504@gmail.com hashimoto.hayato.73e@st.kyoto-u.ac.jp ahasimoto@mm.media.kyoto-u.ac.jp forest@i.kyoto-u.ac.jp Abstract In recent years, there has been a surge of interest in automatically describing images or videos in a natural language. These descriptions are useful for image/video search, etc. In this paper, we focus on pro- cedure execution videos, in which a human makes or repairs something and propose a method for generating procedural texts from them. Since available video/text pairs are limited in size, the direct application of end-to-end deep learning is not feasible. Thus we propose to train Faster R-CNN network for object recognition and LSTM for text generation and combine them at run time. We took pairs of recipe and cooking video as an example, generated a recipe from a video, and compared it with the original recipe. The experimental re- sults showed that our method can produce a recipe as accurate as the state-of-the-art scene descriptions. 1 Introduction Massive effort has been done to develop a method for generating text from vision in the field of natural language processing and computer vision. More specifically, there are number of studies on generating captions for given images or videos (Yang et al., 2011; Rohrbach et al., 2013; Li et al., 2015; Donahue et al., 2015; Karpathy and Fei-Fei, 2015; Shetty and Laaksonen, 2016; Johnson et al., 2016). Most of the existing researches for video captioning, however, deal with simple and short videos (Li et al., 2015; Donahue et al., 2015; Shetty and Laaksonen, 2016) such as a ten second video in which a man playing guitar in a park. In this paper, we propose a new problem in this field: generating a procedural text from an execu- tion video such as cooking or machine assembly. The goal is to develop a method that takes video of a chef cooking a dish from ingredients or a me- chanic assembling a machine from parts as the in- put, and outputs a procedural text that helps an- other person reproduce the same product. We also give an initial solution to the problem, taking cooking recipe generation as an example. Because no large scale corpus consisting of re- lated execution video and procedural text is avail- able for now, we divide the problem into two sub- problems, object recognition and text generation, and train two modules independently using differ- ent resources as their training set. Then we com- bine them and search for the best text. The object recognition module is designed to spot the changes in state of progress of the procedure from video and texts are generated at each time. Finally, some of the generated sentences are selected to cover the entire procedure with discarding redundant sen- tences. In the experiments, we use KUSK Dataset (Hashimoto et al., 2014), which consists …

Content truncated to 500 words