Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows

Proceedings of the 29th International Conference on Computational Linguistics, pages 3570–3577 October 12–17, 2022. 3570 Visual Recipe Flow: A Dataset for Learning Visual State Changes of Objects with Recipe Flows Keisuke Shirai1 Atsushi Hashimoto2 Taichi Nishimura1 Hirotaka Kameko1 Shuhei Kurita3 Yoshitaka Ushiku2 Shinsuke Mori1 1Kyoto University 2OMRON SINIC X Corporation 3RIKEN AIP, JST PRESTO {shirai.keisuke.64x,nishimura.taichi.43x}@st.kyoto-u.ac.jp {atsushi.hashimoto,yoshitaka.ushiku}@sinicx.com {kameko,forest}@i.kyoto-u.ac.jp shuhei.kurita@riken.jp Abstract We present a new multimodal dataset called Visual Recipe Flow, which enables us to learn each cooking action result in a recipe text. The dataset consists of object state changes and the workflow of the recipe text. The state change is represented as an image pair, while the work- flow is represented as a recipe flow graph (r- FG). The image pairs are grounded in the r-FG, which provides the cross-modal relation. With our dataset, one can try a range of applications, from multimodal commonsense reasoning and procedural text generation. 1 Introduction Our aim is to track how foods are processed and changed toward the final food product by each cooking action given a recipe text. This requires some knowledge of the actions: what foods and actions are involved and how the action changes them. Skilled chefs can easily imagine these action effects while understanding the required foods. We are interested in building an autonomous agent en- dowed with this ability, as illustrated in Figure 1. This example involves two cooking actions, and the agent imagines the second action result: the shredded cabbage in the bowl. This also implicates the food requirement: the shredded cabbage pro- duced by the previous action. The prediction for the required foods and action results is indeed a nat- ural ability for humans when they cook something. Thus, this is also crucial for intelligent autonomous agents to understand recipe texts. Previous work on this line of research pro- vided visual annotation for each cooking instruc- tion (Nishimura et al., 2020; Pan et al., 2020). Nishimura et al. (2020) attached an image with bounding boxes of objects to each instruction, while Pan et al. (2020) split an instruction into sentences and attached frames to each sentence. However, their annotations are often insufficient to predict the action result for each object. A typical Figure 1: Our goal is to build an agent that tracks object state changes and predicts what observations can be obtained by cooking actions. case is an instruction in a sentence that directs mul- tiple actions. For example, the instruction of “slice the tomato and put it into the bowl” produces two action results: the sliced tomato and that put in the bowl. Therefore, an instruction-wise visual anno- tation is insufficient for our task, and action-wise visual annotation is required. Preparing a more dense visual annotation is one straightforward way to handle this case. Toward the realization of an agent that predicts the result of each action, we introduce a new mul- timodal dataset called Visual Recipe Flow (VRF). The dataset consists of object1 state changes caused by every action and the workflow of the …

Content truncated to 500 words