Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4525 - 4542 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Understanding Multimodal Procedural Knowledge by Sequencing Multimodal Instructional Manuals Te-Lin Wu1, Alex Spangher2, Pegah Alipoormolabashi3, Marjorie Freedman2, Ralph Weischedel2, Nanyun Peng1 1University of California, Los Angeles, 2ISI, University of Southern California, 3Sharif University of Technology {telinwu,violetpeng}@cs.ucla.edu, palipoor976@gmail.com {spangher,mrf,weisched}@isi.edu Abstract The ability to sequence unordered events is ev- idence of comprehension and reasoning about real world tasks/procedures. It is essential for applications such as task planning and multi-source instruction summarization. It of- ten requires thorough understanding of tem- poral common sense and multimodal infor- mation, since these procedures are often con- veyed by a combination of texts and im- ages. While humans are capable of reason- ing about and sequencing unordered procedu- ral instructions, the extent to which the cur- rent machine learning methods possess such capability is still an open question. In this work, we benchmark models’ capability of rea- soning over and sequencing unordered multi- modal instructions by curating datasets from online instructional manuals and collecting comprehensive human annotations. We find current state-of-the-art models not only per- form significantly worse than humans but also seem incapable of efficiently utilizing multi- modal information. To improve machines’ performance on multimodal event sequenc- ing, we propose sequence-aware pretraining techniques exploiting the sequential alignment properties of both texts and images, resulting in >5% improvements on perfect match ratio. 1 Introduction Instructions are essential sources for agents to learn how to complete complex tasks composed of multi- ple steps (e.g., “making a wood sign from scratch”). However, instructions do not always come in a proper sequential order, for example, when instruc- tions must be combined across sources (e.g., to accomplish a complex task there might be multi- ple useful resources for certain task-steps come out from a single Google search). Therefore, se- quencing unordered task-steps is crucial for com- prehending and inferring task procedures, which requires thorough understanding of event causal and temporal common sense. It is essential for Sand the wood block. Press a sandpaper on and rub it. Disambiguate Apply a wood primer to the surface. Apply the paint to the wood surface if you want bold, solid colors. Use long strokes to apply a wood stain for a more rustic look. Potential Ambiguity Sand the wood block. Press a sandpaper on and rub it. Apply a wood primer to the surface. Apply the paint to the wood surface if you want bold, solid colors. Use long strokes to apply a wood stain for a more rustic look. How To Make A Wood Sign Figure 1: Multimodal task procedure sequencing: The left column shows unordered instruction steps from the man- ual How To Make Wood Signs. Each step is a text description and its associated image. Without the complementary infor- mation from the visuals, a novice may have difficulty inferring the proper task order. Considering multimodal information, the proper order can be correctly inferred …

Content truncated to 500 words