Storyboarding of Recipes: Grounded Contextual Generation

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6040–6046 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6040 Storyboarding of Recipes: Grounded Contextual Generation Khyathi Raghavi Chandu Eric Nyberg Alan W Black Language Technologies Institute, Carnegie Mellon University {kchandu, ehn, awb}@cs.cmu.edu Abstract Information need of humans is essentially multimodal in nature, enabling maximum ex- ploitation of situated context. We introduce a dataset for sequential procedural (how-to) text generation from images in cooking domain. The dataset consists of 16,441 cooking recipes with 160,479 photos associated with different steps. We setup a baseline motivated by the best performing model in terms of human eval- uation for the Visual Story Telling (ViST) task. In addition, we introduce two models to in- corporate high level structure learnt by a Fi- nite State Machine (FSM) in neural sequen- tial generation process by: (1) Scaffolding Structure in Decoder (SSiD) (2) Scaffolding Structure in Loss (SSiL). Our best perform- ing model (SSiL) achieves a METEOR score of 0.31, which is an improvement of 0.6 over the baseline model. We also conducted human evaluation of the generated grounded recipes, which reveal that 61% found that our proposed (SSiL) model is better than the baseline model in terms of overall recipes. We also discuss analysis of the output highlighting key impor- tant NLP issues for prospective directions. 1 Introduction Interpretation is heavily conditioned on context. Real world interactions provide this context in multiple modalities. In this paper, the context is derived from vision and language. The descrip- tion of a picture changes drastically when seen in a sequential narrative context. Formally, this task is defined as: given a sequence of images I = {I1, I2, ..., In} and pairwise associated textual descriptions, T = {T1, T2, ..., Tn}; for a new se- quence I ′, our task is to generate the corresponding T ′. Figure 1 depicts an example for making veg- etable lasagna, where the input is the first row and the output is the second row. We call this a ‘story- board’, since it unravels the most important steps of a procedure associated with corresponding nat- ural language text. The sequential context differ- Lasagna ingredients: tomato sauce or canned tomatoes for making sauce - at least 4-6 cups, one box no boil lasagna noodles, one zucchini, one yellow squash, one jalapeno (or bell pepper!), 1/2 an onion, pinch of oregano, pinch of basil. To make the sauce, cook diced onion in olive oil, and then add the ground beef, garlic and tomato paste. Stir until fragrant and then meat starts to brown and break up, and then add the crushed tomatoes. Pour some water into tomato can and swish it around and then pour that into the pot. Stir well and let simmer while the veg continue to lose moisture. Spoon your ricotta into a bowl and add a good pinch of Italian seasoning and crushed red pepper. I like to add a little black pepper too. …

Content truncated to 500 words