Learning Web-based Procedures by Reasoning over Explanations and Demonstrations in Context

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7652–7662 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 7652 Learning Web-based Procedures by Reasoning over Explanations and Demonstrations in Context Shashank Srivastava1∗ Oleksandr Polozov2 Nebojsa Jojic2 Christopher Meek2 1University of North Carolina, Chapel Hill 2Microsoft Research, Redmond ssrivastava@cs.unc.edu {polozov, jojic, meek}@microsoft.com Abstract We explore learning web-based tasks from a human teacher through natural language expla- nations and a single demonstration. Our ap- proach investigates a new direction for seman- tic parsing that models explaining a demon- stration in a context, rather than mapping explanations to demonstrations. By leverag- ing the idea of inverse semantics from pro- gram synthesis to reason backwards from ob- served demonstrations, we ensure that all con- sidered interpretations are consistent with ex- ecutable actions in any context, thus simplify- ing the problem of search over logical forms. We present a dataset of explanations paired with demonstrations for web-based tasks. Our methods show better task completion rates than a supervised semantic parsing baseline (40% relative improvement on average), and are competitive with simple exploration-and- demonstration based methods, while requiring no exploration of the environment. In learn- ing to align explanations with demonstrations, basic properties of natural language syntax emerge as learned behavior. This is an interest- ing example of pragmatic language acquisition without any linguistic annotation. 1 Introduction People routinely perform repetitive web-based tasks, involving sequences of clicking and typing actions. These include activities such as forwarding emails, booking flight tickets, ordering pizza, etc. These activities largely consist of small sequences of actions in an environment with restricted seman- tics, and are potentially amenable to automation. In this work, we explore whether an AI agent can be taught such tasks through natural language ex- planations and a single demonstration by a user (as one might teach such a task to a human assistant). ∗*Work done while the first author was at Microsoft Re- search. Type ‘news.com’ in the URL bar in the browser, and press enter Type ‘NLP’ in the search at the top-right, and press enter Email me the link to the three most recent articles nlp Date Date Send me NLP news everyday at 8am Step 1: Step 2: Step 3: Can you show me how? Let me teach you … Figure 1: AI assistants that can be taught web-based proce- dures by their users can have diverse practical applications. Here, we explore learning very simple tasks from the Mini World-of-Bits framework using natural language explanations and a single demonstration of the task From the perspective of language understand- ing, this involves challenges such as converting instructional language to actions, resolving ambi- guities through pragmatics, and learning script-like behavior. The web domain is rich in textual, struc- tural and spatial features, allowing for exploration of multiple types of grounding behavior including spatial and visual language understanding, as well as reasoning over semi-structured data. Also, de- spite its richness, the tasks involved usually do not require much background knowledge. From …

Content truncated to 500 words