Improved Instruction Ordering in Recipe-Grounded Conversation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10086–10104 July 9-14, 2023 ©2023 Association for Computational Linguistics Improved Instruction Ordering in Recipe-Grounded Conversation Duong Minh Le, Ruohao Guo, Wei Xu, Alan Ritter Georgia Institute of Technology {dminh6, rguo48}@gatech.edu; {wei.xu, alan.ritter}@cc.gatech.edu Abstract In this paper, we study the task of instructional dialogue and focus on the cooking domain. Analyzing the generated output of the GPT-J model, we reveal that the primary challenge for a recipe-grounded dialog system is how to pro- vide the instructions in the correct order. We hypothesize that this is due to the model’s lack of understanding of user intent and inability to track the instruction state (i.e., which step was last instructed). Therefore, we propose to explore two auxiliary subtasks, namely User Intent Detection and Instruction State Track- ing, to support Response Generation with im- proved instruction grounding. Experimenting with our newly collected dataset, ChattyChef, shows that incorporating user intent and instruc- tion state information helps the response gen- eration model mitigate the incorrect order is- sue. Furthermore, to investigate whether Chat- GPT has completely solved this task, we an- alyze its outputs and find that it also makes mistakes (10.7% of the responses), about half of which are out-of-order instructions. We will release ChattyChef to facilitate further re- search in this area at: https://github. com/octaviaguo/ChattyChef. 1 Introduction Historically, work on conversational agents has mostly fallen into one of two categories: open- domain chatbots (Ritter et al., 2011; Li et al., 2016; Thoppilan et al., 2022; Shuster et al., 2022) or goal- directed dialogue systems within narrow domains (Williams et al., 2016; Eric et al., 2020). However, recent advances in large language models have paved the way for the exploration of dialog agents that can engage in conversations with users to ac- complish open-ended objectives, such as learning about a new topic (Dinan et al., 2019; Choi et al., 2018; Reddy et al., 2019), interpreting bureaucratic policies to answer questions (Saeidi et al., 2018), or negotiating within strategy games (Lewis et al., 2017; Bakhtin et al., 2022). Correct Response 49.6% Incorrect Response Wrong order 22.9% Irrelevant response 10.7% Lack of information 8.4% Wrong information 8.4% Table 1: Manual analysis of 10 recipe-grounded conver- sations (131 responses in total) generated by a fine-tuned GPT-J model on the test portion of our new dataset ChattyChef. The incorrect responses are classified into four error types (examples in Figure 1) with out-of-order instructions being the most common. In this paper, we explore the task of Recipe- Grounded Conversation, where the dialogue agent is expected to converse with a user to walk him/her through the cooking procedure of a recipe, while answering any questions that might arise along the way (see examples in Figure 1). Although many types of dialogue tasks have been proposed and explored, very little prior work has focused on pro- viding instructions to a user to complete a task. In contrast to other dialogue tasks, such as document- grounded …

Content truncated to 500 words