A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4871–4884 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4871 A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks Angela S. Lin ♣∗ Sudha Rao♦ Asli Celikyilmaz♦ Elnaz Nouri♦ Chris Brockett♦ Debadeepta Dey♦ Bill Dolan♦ ♣Salesforce Research, Palo Alto, CA, USA ♦Microsoft Research, Redmond, WA, USA angela.lin@salesforce.com {sudhra, aslicel, elnouri}@microsoft.com {chrisbkt, dedey, billdol}@microsoft.com Abstract Many high-level procedural tasks can be de- composed into sequences of instructions that vary in their order and choice of tools. In the cooking domain, the web offers many partially-overlapping text and video recipes (i.e. procedures) that describe how to make the same dish (i.e. high-level task). Aligning instructions for the same dish across different sources can yield descriptive visual explana- tions that are far richer semantically than con- ventional textual instructions, providing com- monsense insight into how real-world proce- dures are structured. Learning to align these different instruction sets is challenging be- cause: a) different recipes vary in their or- der of instructions and use of ingredients; and b) video instructions can be noisy and tend to contain far more information than text in- structions. To address these challenges, we first use an unsupervised alignment algorithm that learns pairwise alignments between in- structions of different recipes for the same dish. We then use a graph algorithm to de- rive a joint alignment between multiple text and multiple video recipes for the same dish. We release the MICROSOFT RESEARCH MUL- TIMODAL ALIGNED RECIPE CORPUS1 con- taining ∼150K pairwise alignments between recipes across 4,262 dishes with rich common- sense information. 1 Introduction Although machine learning has seen tremendous recent success in challenging game environments such as Go (Schrittwieser et al., 2019), DOTA (Ope- nAI, 2019), and StarCraft (DeepMind, 2019), we have not seen similar progress toward algorithms that might one day help humans perform everyday tasks like assembling furniture, applying makeup, ∗Work done when the author was an intern at Microsoft. 1https://github.com/microsoft/ multimodal-aligned-recipe-corpus 7. Add 12 ounces of thawed peas and bean sprouts. 3. Add onion, garlic, peas and carrots. 4. Transfer shrimp to the hot skillet and cook them one minute per side. 4. Stir fry until tender. 1. Hi everyone. Today we’re making shrimp fried rice, a family favorite. 2. In a small bowl beat together 4 eggs. 2. Heat cooking fat in a large skillet on medium heat. 3. Place a large nonstick pan or wok over medium high heat and when a bead of water sizzles and evaporates, add 2 tablespoons of sesame oil. 1. In a pot add 1 cup of rice and 2 cups of water cook for 15 min. 5. Crack an egg and scramble it in the same pan and mix it throughout vegetables. 6. Add rice and shrimp stir well and remove from heat and add soy sauce. 5. In the same pan cook the beaten eggs breaking them up with your spatula and cooking just until they are no longer running. …

Content truncated to 500 words