Learning Action Conditions from Instructional Manuals for Instruction Understanding

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 3023–3043 July 9-14, 2023 ©2023 Association for Computational Linguistics Learning Action Conditions from Instructional Manuals for Instruction Understanding Te-Lin Wu1, Caiqi Zhang2, Qingyuan Hu1, Alex Spangher3, Nanyun Peng1 1University of California, Los Angeles, 2University of Cambridge, 3Information Sciences Institute, University of Southern California {telinwu,violetpeng,hu528}@cs.ucla.edu, cz391@cam.ac.uk, spangher@isi.edu Abstract The ability to infer pre- and postconditions of an action is vital for comprehending complex instructions, and is essential for applications such as autonomous instruction-guided agents and assistive AI that supports humans to per- form physical tasks. In this work, we propose a task dubbed action condition inference, which extracts mentions of preconditions and postcon- ditions of actions in instructional manuals. We propose a weakly supervised approach utiliz- ing automatically constructed large-scale train- ing instances from online instructions, and cu- rate a densely human-annotated and validated dataset to study how well the current NLP mod- els do on the proposed task. We design two types of models differ by whether contextual- ized and global information is leveraged, as well as various combinations of heuristics to construct the weak supervisions. Our exper- iments show a >20% F1-score improvement with considering the entire instruction contexts and a > 6% F1-score benefit with the proposed heuristics. However, the best performing model is still well-behind human performance.1 1 Introduction When performing complex tasks (e.g. making a gourmet dish), instructional manuals are often re- ferred to as useful guidelines. To follow the in- structed actions, it is crucial to understand the pre- conditions, i.e. prerequisites before taking a partic- ular action, and the postconditions, i.e. the status supposed to be reached after performing the ac- tion. Knowledge of action-condition dependencies is prevalent and inferable in many instructional texts. For example, in Figure 1, before performing the action “place onions" in step 3, both precondi- tions: “heat the pan" (in step 2) and “slice onions" (in step 1) have to be successfully accomplished. Likewise, executing “stir onions" (in step 4), leads to its postcondition, “caramelized" (also in step 4). 1Dataset and codes will be released at: here. 1. Slice 500 grams of onion. 2. Heat the pan with olive oil. Wait until the oil is sizzling. 3. Place onions in the frying pan. 4. Stir the onions. In a few minutes, they should be caramelized. Precondition Postconditions Precondition Precondition Postcondition Can I put the onions in now? Yes, your pan is already fried, onions are also sliced. Figure 1: The Action Condition Inference Task: We pro- pose a task that probes models’ ability to infer both precon- ditions and postconditions of an action from instructional manuals. It has wide applications to e.g. assistive AI and task- solving robots. ∗This instruction is simplified for illustration. For autonomous agents or assistant AI that aids humans to accomplish tasks, understanding the con- ditions provides a structured view of a task (Linden, 1994; Aeronautiques et al., 1998; Branavan et al., 2012a; Sharma and Kroemer, 2020) and …

Content truncated to 500 words