Reasoning about Goals, Steps, and Temporal Ordering with WikiHow

Reasoning about Goals, Steps, and Temporal Ordering with WikiHow Li Zhang∗ Qing Lyu∗ University of Pennsylvania {lyuqing,zharry,ccb}@seas.upenn.edu Chris Callison-Burch Abstract We propose a suite of reasoning tasks on two types of relations between procedural events: GOAL-STEP relations (“learn poses” is a step in the larger goal of “doing yoga”) and STEP- STEP TEMPORAL relations (“buy a yoga mat” typically precedes “learn poses”). We intro- duce a dataset targeting these two relations based on wikiHow, a website of instructional how-to articles. Our human-validated test set serves as a reliable benchmark for common- sense inference, with a gap of about 10% to 20% between the performance of state-of- the-art transformer models and human perfor- mance. Our automatically-generated training set allows models to effectively transfer to out- of-domain tasks requiring knowledge of pro- cedural events, with greatly improved perfor- mances on SWAG, Snips, and Story Cloze Test in zero- and few-shot settings. 1 Introduction If you ask Alexa or Siri where to “buy a yoga mat,” it should ideally infer that your goal is probably to “do yoga” and therefore suggest information on subsequent steps like “learn some poses.” This re- quires the system to reason about the GOAL-STEP relation and the STEP-STEP TEMPORAL relation among events in a procedure. Though event rela- tion reasoning is a popular task, existing datasets mostly focus on temporal relations (Pustejovsky et al., 2003; Ning et al., 2018), causal relations (Hashimoto et al., 2014; Caselli and Vossen, 2017), spatiotemporal containment and coreference rela- tions (Glavaˇs et al., 2014; Liu et al., 2014). Less attention has been paid to relations among proce- dural events, the understanding of which is critical to task-oriented intelligent systems. The knowl- edge of procedural events is also crucial to learning ∗ Equal contribution. Figure 1: Goals and steps (slightly paraphrased) from wikiHow articles “How to Do Yoga” and “How to Warm Up”. The lines denote GOAL-STEP relations; the arrows denote STEP-STEP TEMPORAL relations. scripts (Feigenbaum et al., 1981), which describe sequences of stereotypical human activities. To bridge this gap, we propose a dataset for goal- step inference targeting these two event relations. We collect data from wikiHow1, a website con- sisting of more than 110,000 professionally-edited how-to articles spanning a surprisingly wide range of domains. Each wikiHow article describes a com- monplace human activity, organized as a goal and a sequence of steps (Figure 1). Our dataset includes 3 tasks: inferring steps given a goal, the goal given a step, and the ordering between two steps given a goal. For each task, we automatically generate 100,000 to 800,000 examples as the training set, us- ing a negative sampling strategy based on semantic similarity; we also provide a human-validated test set with 1,000 to 3,000 examples.2 Our test set serves as a reliable benchmark for commonsense inference, with a performance gap of 10% to 20% between human and state-of-the-art transformer models trained in-domain. Moreover, when pre-trained on our tasks, a model can transfer knowledge of procedural event relations to other NLU tasks, with a …

Content truncated to 500 words