Visual Grounding Annotation of Recipe Flow Graph

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4275–4284 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4275 Visual Grounding Annotation of Recipe Flow Graph Taichi Nishimura1, Suzushi Tomori1, Hayato Hashimoto1, Atsushi Hashimoto2, Yoko Yamakata3, Jun Harashima4, Yoshitaka Ushiku2, Shinsuke Mori5 1Graduate School of Informatics, Kyoto University, 2OMRON SINIC X Corporation 3Graduate School of Information Science and Technology, The University of Tokyo 4Cookpad Inc, 5Academic Center for Computing and Media Studies, Kyoto University 1,5Yoshidahonmachi, Sakyoku, Kyoto, Japan, 606-8501 25-24-5, Hongo, Bunkyo-ku, Tokyo, Japan, 113–0033 37-3-1 Hongo, Bunkyo-ku, Tokyo, Japan, 113–8656 44-20-3 Ebisu, Shibuya-ku, Tokyo, Japan, 150–6012 1{nishimura.taichi.43x,tomori.suzushi.72e,hashimoto.hayato.73e}@st.kyoto-u.ac.jp, 2{atsushi.hashimoto, yoshitaka.ushiku}@sinicx.com, 3yamakata@mi.u-tokyo.ac.jp, 4jun-harashima@cookpad.com, 5forest@i.kyoto-u.ac.jp Abstract In this paper, we provide a dataset that gives visual grounding annotations to recipe flow graphs. A recipe flow graph is a representation of the cooking workflow, which is designed to understand the workflow from natural language processing. Such a workflow will increase its value when grounded to real-world activities, and visual grounding is a way to do so. Visual grounding is provided as bounding boxes to image sequences of recipes, and each bounding box is linked to an element of the workflow. Because the workflows are also linked to the text, this annotation gives visual grounding with workflow’s contextual information between procedural text and visual observation in an indirect manner. We subsidiarily annotated two types of event attributes with each bounding box: “doing-the-action,” or “done-the-action”. As a result of the annotation, we got 2,300 bounding boxes in 272 flow graph recipes. Various experiments showed that the proposed dataset enables us to estimate contextual information described in recipe flow graphs from an image sequence. Keywords: Procedural Text, Bounding Box, Flow Graph, Visual Grounding 1 Introduction Procedural texts are suitable for the target of natural lan- guage understanding (NLU) because they are goal-oriented descriptions and we can almost define the understanding of their goals. Some studies have proposed the framework of understanding procedural texts with graphs, which can represent the entire workflow (Mori et al., 2014; Jermsurawong and Habash, 2014; Kiddon et al., 2015). These frameworks would be helpful for real-world systems, such as smart kitchen (Hashimoto et al., 2008) and cooking robot (Bollini et al., 2013), to understand the context of these workflows and execute next actions. Therefore, it is essential to ground visual observations (images or videos) with procedural texts. Visual grounding is one of the solutions to help computers to understand which objects are aligned with textual descriptions. In the previous studies, they targetted the pair of general texts and visual observations. MSCOCO (Lin et al., 2014), Flickr30k (Plummer et al., 2015), and YouTube-BoundingBox (Real et al., 2017) are typical datasets for such tasks. They reported that annotating bounding boxes with a textual description helps computers to know which objects they have to pay attention to for image captioning (Xu et al., 2015; Cornia et al., 2019), visual question answering (VQA) (Fukui et al., 2016), and some other visual grounding tasks (Huang et al., 2017; …

Content truncated to 500 words