lilGym: Natural Language Visual Reasoning with Reinforcement Learning

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 9214–9234 July 9-14, 2023 ©2023 Association for Computational Linguistics lilGym: Natural Language Visual Reasoning with Reinforcement Learning Anne Wu, Kianté Brantley, Noriyuki Kojima, and Yoav Artzi Department of Computer Science and Cornell Tech, Cornell University {aw588, kdb82, nk654}@cornell.edu, {yoav}@cs.cornell.edu Abstract We present lilGym, a new benchmark for language-conditioned reinforcement learning in visual environments. lilGym is based on 2,661 highly-compositional human-written nat- ural language statements grounded in an in- teractive visual environment. We introduce a new approach for exact reward computa- tion in every possible world state by annotat- ing all statements with executable Python pro- grams. Each statement is paired with multi- ple start states and reward functions to form thousands of distinct Markov Decision Pro- cesses of varying difficulty. We experiment with lilGym with different models and learn- ing regimes. Our results and analysis show that while existing methods are able to achieve non-trivial performance, lilGym forms a chal- lenging open problem. lilGym is available at https://lil.nlp.cornell.edu/lilgym/. 1 Introduction The ability to reason about natural language has the potential to transform how reinforcement learn- ing (RL) is used to train grounded agents. Lan- guage provides an expressive and accessible con- duit for task specification, enabling agents to ad- dress a broad set of tasks, rather than to learn single- behavior policies. At the same time, RL is a promis- ing framework to study core grounded language learning problems within interactive environments. Prerequisite to realizing this potential are expres- sive benchmark environments, as has been instru- mental for progress in RL more broadly. However, natural language poses unique challenges to build- ing such benchmarks. Beyond the design of the en- vironment itself, which must support rich linguistic reasoning, accurate reward computation requires resolving language semantics. Existing approaches adopt different strategies to address this issue, most often by using synthetic language (e.g., Côté et al., 2018; Co-Reyes et al., 2019), or by removing the There are two boxes which has the same number of objects and three kinds of colors. REMOVE(x=142, y=41) ADD(x=142, y=55, ) ADD(x=288, y=42, ) ADD(x=357, y=63, ) Figure 1: An illustrative example of lilGym. Given a natural language statement with a target truth value, True in this example (omitted from the figure), and an initial image, the agent needs to manipulate objects in the environment so that the truth value of the statement with respect to the image is the target truth value. language problem from reward computation by re- stricting to a single goal state (e.g., Anderson et al., 2018; Chen et al., 2019). While these approaches open new research avenues, both have significant drawbacks. The simplifications of synthetic lan- guage limit the relevance of methods and results to the complexities of human language. Single-goal formulations forgo or restrict language’s ability to efficiently abstract over many possible solutions, a core argument for its potential to RL. We present lilGym, an RL benchmark where an agent manipulates a visual environment …

Content truncated to 500 words