Non-Sequential Graph Script Induction via Multimedia Grounding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5529–5545 July 9-14, 2023 ©2023 Association for Computational Linguistics Non-Sequential Graph Script Induction via Multimedia Grounding Yu Zhou1, Sha Li2, Manling Li2, Xudong Lin3, Shih-Fu Chang3, Mohit Bansal4, Heng Ji2 1 University of California, Los Angeles 2 University of Illinois Urbana-Champaign 3 Columbia University 4 University of North Carolina at Chapel Hill yu.zhou@ucla.edu, {shal2, manling2, hengji}@illinois.edu mbansal@cs.unc.edu, {xudong.lin, shih.fu.chang}@columbia.edu Abstract Online resources such as wikiHow compile a wide range of scripts for performing everyday tasks, which can assist models in learning to reason about procedures. 1 However, the scripts are always presented in a linear manner, which does not reflect the flexibility displayed by peo- ple executing tasks in real life. For example, in the CrossTask Dataset, 64.5% of consecutive step pairs are also observed in the reverse order, suggesting their ordering is not fixed. In addi- tion, each step has an average of 2.56 frequent2 next steps, demonstrating "branching". In this paper, we propose a new challenging task of non-sequential graph script induction, aiming to capture optional and interchangeable steps in procedural planning. To automate the induc- tion of such graph scripts for given tasks, we propose to take advantage of loosely aligned videos of people performing the tasks. In par- ticular, we design a multimodal framework to ground procedural videos to wikiHow textual steps and thus transform each video into an observed step path on the latent ground truth graph script. This key transformation enables us to train a script knowledge model capable of both generating explicit graph scripts for learnt tasks and predicting future steps given a partial step sequence. Our best model outperforms the strongest pure text/vision baselines by 17.52% absolute gains on F1@3 for next step prediction and 13.8% absolute gains on Acc@1 for partial sequence completion. Human evaluation shows our model outperforming the wikiHow linear baseline by 48.76% absolute gains in capturing sequential and non-sequential step relations. 1 Introduction A script consists of typical actions that are per- formed to complete a given task. Online re- 1Our data and code are publicly available for re- search purposes at https://github.com/bryanzhou008/ Multimodal-Graph-Script-Learning/ 2Occurred in more than 10 videos. Non-sequential Graph Script 0.48 Add some salt 0.72 0.43 Put in cooked eggs 0.75 Add chopped green onions Mix well and serve 0.65 Cook cold rice in the wok for 3min 0.52 Pre-cook some eggs 0.43 Cut some green onions 0.37 Add some chilli peppers 0.27 0.25 0.79 Heat the wok with oil until hot Mix well and serve Put in 2 red hot chilli peppers Put in cold rice and cook 3min Heat the wok for 3 minutes Put in the cooked eggs Add green onions Add a touch of salt Chop up some green onions Pre-cook 3 whole eggs Figure 1: Example of wikiHow linear script of the pro- cedural task make egg fried rice compared to an ideal example of our non-sequential graph script consisting of optional and …

Content truncated to 500 words