Distilling Script Knowledge from Large Language Models for Constrained Language Planning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 4303–4325 July 9-14, 2023 ©2023 Association for Computational Linguistics Distilling Script Knowledge from Large Language Models for Constrained Language Planning Siyu Yuan♡, Jiangjie Chen♠∗, Ziquan Fu♣†, Xuyang Ge♠, Soham Shah♢, Charles Robert Jankowski♢, Yanghua Xiao♠¶, Deqing Yang♡∗ ♡School of Data Science, Fudan University ♠Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University ♣System Inc. ♢Brain Technologies, Inc. ¶Fudan-Aishu Cognitive Intelligence Joint Research Center syyuan21@m.fudan.edu.cn, jjchen19@fudan.edu.cn frank@system.com, {sshah,cjankowski}@brain.im {xyge20,shawyh,yangdeqing}@fudan.edu.cn Abstract In everyday life, humans often plan their ac- tions by following step-by-step instructions in the form of goal-oriented scripts. Previous work has exploited language models (LMs) to plan for abstract goals of stereotypical activities (e.g., “make a cake”), but leaves more specific goals with multi-facet constraints understudied (e.g., “make a cake for diabetics”). In this pa- per, we define the task of constrained language planning for the first time. We propose an over- generate-then-filter approach to improve large language models (LLMs) on this task, and use it to distill a novel constrained language plan- ning dataset, CoScript, which consists of 55,000 scripts. Empirical results demonstrate that our method significantly improves the con- strained language planning ability of LLMs, especially on constraint faithfulness. Further- more, CoScript is demonstrated to be quite effective in endowing smaller LMs with con- strained language planning ability. 1 1 Introduction To accomplish everyday goals, humans usually plan their actions in accordance with step-by-step instructions. Such instructions are discovered as goal-oriented scripts (Schank and Abelson, 1975, 2013), involving a set of prototypical event se- quences to achieve goals. For the example in Fig- ure 1, to achieve the goal (make a cake), one usu- ally has to follow certain steps of instructions, e.g., gather ingredients, preheat the oven, etc. The plan- ning for such step-by-step scripts chains up rea- soning toward complex goals (Abelson, 1976; Wei et al., 2022). Therefore, the automation of plan- ning envisions more intelligent and reasonable AI ∗Corresponding authors. †Work done while at Brain Technologies, Inc. 1Resources of this paper can be found at https:// github.com/siyuyuan/coscript. 1) Gather your ingredients. 2) Preheat oven to 350 °F. Grease an 8-inch (20 cm) cake pan with butter or margarine. 3) In a medium bowl, combine flour, baking powder and salt. 4) In a large bowl, cream butter or margarine and sugar together until light and fluffy. Beat in eggs, one at a time. 5) Add vanilla extract and almond extract. 6) Gradually add flour mixture to the large bowl, mixing until just blended. 7) Pour batter into prepared cake pan. 8) Bake cake for 30 minutes. 9) Remove cake from oven and let cool before serving. Script for the Goal: Make a Cake for Diabetics Sugar for diabetics?! InstructGPT (text-davinci-002) Steps Figure 1: A list of steps InstructGPT generates to plan for the goal “make a cake for diabetics”. InstructGPT mistakenly adds sugar to the cake, which is unfit for diabetic patients. This example shows that …

Content truncated to 500 words