Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 1179 - 1189 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension Huibin Zhang1, Zhengkun Zhang1, Yao Zhang1, Jun Wang2∗, Yufan Li1 Ning Jiang3, Xin Wei3, Zhenglu Yang1∗ 1TKLNDST, CS, Nankai University, China, 2Ludong University, China, 3Mashang Consumer Finance Co., Ltd., China {zhanghuibin.cc,zhangzk2017,yaozhang,junwang,1811486} @mail.nankai.edu.cn,{ning.jiang02,xin.wei02}@msxf.com, yangzl@nankai.edu.cn Abstract Procedural Multimodal Documents (PMDs) organize textual instructions and correspond- ing images step by step. Comprehending PMDs and inducing their representations for the downstream reasoning tasks is designated as Procedural MultiModal Machine Compre- hension (M3C). In this study, we approach Pro- cedural M3C at a fine-grained level (compared with existing explorations at a document or sentence level), that is, entity. With delicate consideration, we model entity both in its tem- poral and cross-modal relation and propose a novel Temporal-Modal Entity Graph (TMEG). Specifically, a heterogeneous graph structure is formulated to capture textual and visual en- tities and trace their temporal-modal evolution. In addition, a graph aggregation module is in- troduced to conduct graph encoding and rea- soning. Comprehensive experiments across three Procedural M3C tasks are conducted on a traditional dataset RecipeQA and our new dataset CraftQA, which can better evaluate the generalization of TMEG. 1 Introduction MultiModal Machine Comprehension (M3C) is a generalization of machine reading compre- hension by introducing multimodality. Due to its differences from Visual Question Answering (VQA) (Antol et al., 2015) in the form of under- standing multimodal contexts and conducting mul- timodal questions and answers, there has been a lot of attention in recent years devoted to this field. In this paper, we investigate a task that has been typical of M3C recently, named Procedural M3C, a task of reading comprehension of Procedural Multimodal Documents (PMDs). As shown in Figure 1, a recipe that contains suc- cessive multimodal instructions is a typical PMD. Reading a recipe seems trivial for humans but is still complex for a machine reading comprehension *Corresponding authors. Step 4: Mandarine Oranges: Drain them. Blue- berries: Wash them. Strawberries: Wash them. Cut off the stems. Kiwifruit: Wash them. Go all the way through and gently ... Step 2: Make or buy your sugar cookie dough. Cut into cookie sizes and bake according to your directions. Step 1: Ingredients 16 oz Strawberry, 9 oz Blue- berries, 11 oz can of Mandarine Oranges, 4-5 Kiwifruits, Sugar Cookie Dough, 2 tsp. Vanilla, 1/2 c. Sugar, 1 pkg. Cream Cheese - softened. Supplies: ... … … Image Text Temporal Relation Multimodal Relation Entity Figure 1: A PMD instance: a recipe about making mini fruit pizza. The entities and relations (i.e. temporal and multimodal) among them are highlighted. system before it can comprehend both textual and visual contents and capture their relations. Current Procedural M3C studies (Yagcioglu et al., 2018; Liu et al., 2020) comprehend PMDs by encoding text and images at each procedure step. These efforts, however, only scratch the surface and lack deep insight into the …

Content truncated to 500 words