Hey AI, Can You Solve Complex Tasks by Talking to Agents?

Findings of the Association for Computational Linguistics: ACL 2022, pages 1808 - 1823 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Hey AI, Can You Solve Complex Tasks by Talking to Agents? Tushar Khot Kyle Richardson Daniel Khashabi Ashish Sabharwal Allen Institute for AI, Seattle, WA, U.S.A. {tushark,kyler,danielk,ashishs}@allenai.org Abstract Training giant models from scratch for each complex task is resource- and data-inefficient. To help develop models that can leverage ex- isting systems, we propose a new challenge: Learning to solve complex tasks by communi- cating with existing agents (or models) in nat- ural language. We design a synthetic bench- mark, COMMAQA, with three complex reason- ing tasks (explicit, implicit, numeric) designed to be solved by communicating with existing QA agents. For instance, using text and table QA agents to answer questions such as "Who had the longest javelin throw from USA?". We show that black-box models struggle to learn this task from scratch (accuracy under 50%) even with access to each agent’s knowledge and gold facts supervision. In contrast, mod- els that learn to communicate with agents out- perform black-box models, reaching scores of 100% when given gold decomposition supervi- sion. However, we show that the challenge of learning to solve complex tasks by communi- cating with existing agents without relying on any auxiliary supervision or data still remains highly elusive. We release COMMAQA, along with a compositional generalization test split, to advance research in this direction.1 1 Introduction A common research avenue pursued these days is to train monolithic language models with billions of parameters (Radford et al., 2019; Raffel et al., 2020; Brown et al., 2020) to solve every language un- derstanding and reasoning challenge (Wang et al., 2018, 2019). In contrast, humans often tackle com- plex tasks by breaking them down into simpler sub- tasks, and solving these by interacting with other people or automated agents whose skill-sets we are familiar with. This approach allows us to learn to solve new complex tasks quickly and effectively, by building upon what’s already known. Can AI systems learn to do the same? 1https://github.com/allenai/commaqa Added Alexa, Buy the book "Harry Potter and the Sorcerer’s Stone" Hey Google, which book series has a kid with a lightning scar? Harry Potter Hey Google, what are the books in Harry Potter series? 1. Harry Potter and the Sorcerer’s Stone 2. Harry Potter and the Chamber of Secrets ... Alexa, Buy the book "Harry Potter and the Sorcerer’s Stone" Added Buy the entire book series with the kid with the lightning scar Figure 1: Motivating example for a setup where a sys- tem is expected to learn to accomplish goals by inter- acting with agents via a natural language interface. To facilitate research in this direction, we pro- pose a new reasoning challenge and a benchmark called COMMAQA where, in addition to the usual end-task supervision, one has access to a set of pre- defined AI agents with examples of their natural language inputs. Importantly, the target end-task is designed to be too difficult …

Content truncated to 500 words