Procedural Reasoning Networks for Understanding Multimodal Procedures

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 441–451 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 441 Procedural Reasoning Networks for Understanding Multimodal Procedures Mustafa Sercan Amac Semih Yagcioglu Aykut Erdem Erkut Erdem Hacettepe University Computer Vision Lab Dept. of Computer Engineering, Hacettepe University, Ankara, TURKEY {b21626915,n13242994,aykut,erkut}@cs.hacettepe.edu.tr Abstract This paper addresses the problem of com- prehending procedural commonsense knowl- edge. This is a challenging task as it re- quires identifying key entities, keeping track of their state changes, and understanding tem- poral and causal relations. Contrary to most of the previous work, in this study, we do not rely on strong inductive bias and explore the question of how multimodality can be ex- ploited to provide a complementary semantic signal. Towards this end, we introduce a new entity-aware neural comprehension model aug- mented with external relational memory units. Our model learns to dynamically update en- tity states in relation to each other while read- ing the text instructions. Our experimental analysis on the visual reasoning tasks in the recently proposed RecipeQA dataset reveals that our approach improves the accuracy of the previously reported models by a large margin. Moreover, we find that our model learns effec- tive dynamic representations of entities even though we do not use any supervision at the level of entity states.1 1 Introduction A great deal of commonsense knowledge about the world we live is procedural in nature and involves steps that show ways to achieve specific goals. Un- derstanding and reasoning about procedural texts (e.g. cooking recipes, how-to guides, scientific pro- cesses) are very hard for machines as it demands modeling the intrinsic dynamics of the procedures (Bosselut et al., 2018; Dalvi et al., 2018; Yagcioglu et al., 2018). That is, one must be aware of the entities present in the text, infer relations among them and even anticipate changes in the states of the entities after each action. For example, consider the cheeseburger recipe presented in Fig. 1. The 1The project website with code and demo is available at https://hucvl.github.io/prn/ instruction “salt and pepper each patty and cook for 2 to 3 minutes on the first side” in Step 5 entails mixing three basic ingredients, the ground beef, salt and pepper, together and then applying heat to the mix, which in turn causes chemical changes that alter both the appearance and the taste. From a natural language understanding perspective, the main difficulty arises when a model sees the word patty again at a later stage of the recipe. It still cor- responds to the same entity, but its form is totally different. Over the past few years, many new datasets and approaches have been proposed that address this in- herently hard problem (Bosselut et al., 2018; Dalvi et al., 2018; Tandon et al., 2018; Du et al., 2019). To mitigate the aforementioned challenges, the ex- isting works rely mostly on heavy supervision and focus on predicting the individual state changes of entities at each step. Although these …

Content truncated to 500 words