ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments

Proceedings of the 29th International Conference on Computational Linguistics, pages 5597–5612 October 12–17, 2022. 5597 ACT-Thor: A Controlled Benchmark for Embodied Action Understanding in Simulated Environments Michael Hanna∗† University of Amsterdam m.w.hanna@uva.nl Federico Pedeni∗ University of Trento federico.pedeni @studenti.unitn.it Alessandro Suglia Heriot-Watt University a.suglia@hw.ac.uk Alberto Testoni University of Trento alberto.testoni@unitn.it Raffaella Bernardi University of Trento raffaella.bernardi@unitn.it Abstract Artificial agents are nowadays challenged to perform embodied AI tasks. To succeed, agents must understand the meaning of verbs and how their corresponding actions transform the sur- rounding world. In this work, we propose ACT-Thor, a novel controlled benchmark for embodied action understanding. We use the AI2-THOR simulated environment to produce a controlled setup in which an agent, given a before-image and an associated action com- mand, has to determine what the correct after- image is among a set of possible candidates. First, we assess the feasibility of the task via a human evaluation that resulted in 81.4% accu- racy, and very high inter-annotator agreement (84.9%). Second, we design both unimodal and multimodal baselines, using state-of-the- art visual feature extractors. Our evaluation and error analysis suggest that only models that have a very structured representation of the ac- tions together with powerful visual features can perform well on the task. However, they still fall behind human performance in a zero-shot scenario where the model is exposed to unseen (action, object) pairs. This paves the way for a systematic way of evaluating embodied AI agents that understand grounded actions. 1 Introduction Recently, embodied agents have been increasingly proposed and evaluated on their capacity to nav- igate virtual environments, and execute actions within them according to instructions (Suhr et al., 2019; Hahn et al., 2020; Shridhar et al., 2020; Padmakumar et al., 2022). Action execution as a means of evaluating language understanding is a long-standing idea (e.g. (Winograd, 1972)) which has now become feasible thanks to the release *Equal contribution †Work performed while at the University of Trento Figure 1: The image on the left (before-image) is the view the robot has while holding a plate or a cup; the image on the right is the view the robot has after having executed the command of picking up that plate/cup. of various virtual environment platforms (Kolve et al., 2017; Savva et al., 2019; Yan et al., 2018). Thus, the study of embodied agents is an interesting venue to push forward due its both theoretical and practical appeal. However, fully embodied tasks involve a variety of issues, making it hard to under- stand where we stand with respect to the challenges posed. Here, we focus on one core aspect of such embodied tasks: action grounding. Tamari et al. (2020) advocate for embodied language understanding as a necessary step to overcome the limitations of the mainstream deep- learning paradigm. Among the priorities of their agenda is evaluating models’ ability to simulate the effects of actions. Evidence from neuroscience favors an embodied meaning of words and specif- ically of verbs; for instance, the natural theory of 5598 language …

Content truncated to 500 words