FLIN: A Flexible Natural Language Interface for Web Navigation

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2777–2788 June 6–11, 2021. ©2021 Association for Computational Linguistics 2777 FLIN: A Flexible Natural Language Interface for Web Navigation Sahisnu Mazumder∗ Department of Computer Science University of Illinois at Chicago, USA sahisnumazumder@gmail.com Oriana Riva Microsoft Research Redmond, USA oriana.riva@microsoft.com Abstract AI assistants can now carry out tasks for users by directly interacting with website UIs. Current semantic parsing and slot-filling tech- niques cannot flexibly adapt to many different websites without being constantly re-trained. We propose FLIN, a natural language interface for web navigation that maps user commands to concept-level actions (rather than low-level UI actions), thus being able to flexibly adapt to different websites and handle their transient nature. We frame this as a ranking problem: given a user command and a webpage, FLIN learns to score the most relevant navigation in- struction (involving action and parameter val- ues). To train and evaluate FLIN, we collect a dataset using nine popular websites from three domains. Our results show that FLIN was able to adapt to new websites in a given domain. 1 Introduction AI personal assistants, such as Google Assistant, can now interact directly with the UI of websites to carry out human tasks (Tech Crunch, 2019). Users issue commands to the assistant, and this executes them by typing, selecting items, clicking buttons, and navigating to different pages in the website. Such an approach is appealing as it can reduce the dependency on third-party APIs and expand an as- sistant’s capabilities. This paper focuses on a key component of such systems: a natural language (NL) interface capable of mapping user commands (e.g., “find an Italian restaurant for 7pm") into nav- igation instructions that a web browser can execute. One way to implement such an NL interface is to map user commands directly into low-level UI actions (button clicks, text inputs, etc.). The UI elements appearing in a webpage are embedded by concatenating their DOM attributes (tag, classes, text, etc.). Then, a scoring function (Pasupat et al., 2018) or a neural policy (Liu et al., 2018a) are ∗Work done while interning at Microsoft Research. trained to identify which UI element best supports a given command. Learning at the level of UI elements is effective, but only in controlled (UI el- ements do not change over time (Shi et al., 2017a)) or restricted (single applications (Branavan et al., 2009)) environments. This is not the case in the “real” web, where (i) websites are constantly up- dated, and (ii) a user may ask an assistant to execute the same task in any website of their choice (e.g., ordering pizza with dominos.com or pizzahut.com). The transient nature and diversity of the web call for an NL interface that can flexibly adapt to en- vironments with a variable and unknown set of actions, without being constantly re-trained. To achieve this goal, we take two steps. First, we conceptualize a new way of designing NL inter- …

Content truncated to 500 words