Grounding Open-Domain Instructions to Automate Web Support Tasks

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1022–1032 June 6–11, 2021. ©2021 Association for Computational Linguistics 1022 Grounding Open-Domain Instructions to Automate Web Support Tasks Nancy Xu Computer Science Dept. Stanford University xnancy@stanford.edu Sam Masling Computer Science Dept. Stanford University smasling@stanford.edu Michael Du Computer Science Dept. Stanford University mdu7@stanford.edu Giovanni Campagna Computer Science Dept. Stanford University gcampagn@stanford.edu Larry Heck Viv Labs Samsung Research larry.heck@ieee.org James Landay Computer Science Dept. Stanford University landay@stanford.edu Monica S Lam Computer Science Dept. Stanford University lam@stanford.edu Abstract Grounding natural language instructions on the web to perform previously unseen tasks enables accessibility and automation. We in- troduce a task and dataset to train AI agents from open-domain, step-by-step instructions originally written for people. We build RUSS (Rapid Universal Support Service) to tackle this problem. RUSS consists of two models: First, a BERT-LSTM with pointers parses in- structions to ThingTalk, a domain-specific lan- guage we design for grounding natural lan- guage on the web. Then, a grounding model retrieves the unique IDs of any webpage ele- ments requested in ThingTalk. RUSS may in- teract with the user through a dialogue (e.g. ask for an address) or execute a web operation (e.g. click a button) inside the web runtime. To augment training, we synthesize natural lan- guage instructions mapped to ThingTalk. Our dataset consists of 80 different customer ser- vice problems from help websites, with a to- tal of 741 step-by-step instructions and their corresponding actions. RUSS achieves 76.7% end-to-end accuracy predicting agent actions from single instructions. It outperforms state- of-the-art models that directly map instruc- tions to actions without ThingTalk. Our user study shows that RUSS is preferred by actual users over web navigation. 1 Introduction Grounding natural language is a key to building robots and AI agents (Chen and Mooney, 2011) that interact seamlessly with people. Besides grounding tasks visually (Mirowski et al., 2018; Venugopalan et al., 2015), future AI agents must be able to ground language and execute actions on the web. We build a general-purpose, interactive agent to master tasks from open-domain natural language instructions on websites. We focus on the service domain for tasks such as redeeming a gift card, 23 1. . . Then enter the user’s email in the text field under email address or username Ask the user for their email Go to https://www.amazon.com/wishlist 3. 2. 1. @retrieve (description = "email address or username") => @retrieve (type = enum:input, below = id) => @enter (dict_key = "email", element_id = id) @ask (dict_key = "email") @goto (website = "https://www.amazon.com/wishlist") 3. 2. 1. @enter(dict_key = ”email”, element_id = 37) @ask(dict_key = “email”) @goto(website = https://www.amazon.com/wishlist) Semantic Parser Grounding Model Runtime Step 2: User Interaction -- “What is your email? “ Step 3: Web Action -- (enter, “ann@gmail.com”) Expert Instructions ThingTalk Parses Grounded Actions Active Webpage User Webpage per instruction param:email = “ann@gmail.com” element 37 Figure 1: RUSS’s semantic parser maps natural lan- guage instructions into ThingTalk (our DSL) and …

Content truncated to 500 words