HOLM: Hallucinating Objects with Language Models for Referring Expression Recognition in Partially-Observed Scenes

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 5440 - 5453 May 22-27, 2022 c⃝2022 Association for Computational Linguistics HOLM: Hallucinating Objects with Language Models for Referring Expression Recognition in Partially-Observed Scenes Volkan Cirik1 Louis-Philippe Morency1 Taylor Berg-Kirkpatrick2 1Carnegie Mellon University 2University of California San Diego {vcirik,morency}@cs.cmu.edu {tberg}@eng.ucsd.edu Abstract AI systems embodied in the physical world face a fundamental challenge of partial ob- servability; operating with only a limited view and knowledge of the environment. This cre- ates challenges when AI systems try to reason about language and its relationship with the environment: objects referred to through lan- guage (e.g. giving many instructions) are not immediately visible. Actions by the AI system may be required to bring these objects in view. A good benchmark to study this challenge is Dynamic Referring Expression Recognition (dRER) task where the goal is to find a tar- get location by dynamically adjusting the field of view (FoV) in a partially observed 360◦ scenes. In this paper, we introduce HOLM, Hallucinating Objects with Language Models, to address the challenge of partial observabil- ity. HOLM uses large pre-trained language models (LMs) to infer object hallucinations for the unobserved part of the environment. Our core intuition is that if a pair of objects co- appear in an environment frequently, our us- age of language should reflect this fact about the world. Based on this intuition, we prompt language models to extract knowledge about object affinities which gives us a proxy for spatial relationships of objects. Our experi- ments show that HOLM performs better than the state-of-the-art approaches on two datasets for dRER; allowing to study generalization for both indoor and outdoor settings. 1 Introduction One of the fundamental challenges in building AI systems physically present in the world is ad- dressing the issue of partial observability, the phe- nomenon where the entire state of the environment is not known or available to the system. People cope with partial observability by reasoning about what is not immediately visible (see example in Figure 1). People combine their general knowl- edge about the world and adapt their knowledge ↗ ↗ ↗ ↗ Figure 1: Illustration of our main contribution: Halluci- nating Objects. Knowledge about object relationships is helpful when navigating in an unknown and partially observed environment. In the example above, the TV is not visible, but the couch hints that a TV might be in front of it because usually couches face TVs. to specific contexts (Torralba et al., 2006). Gen- eral knowledge about kitchens can help to know approximately where to look for pans or utensils in a kitchen that has never been seen before. How can an AI system build general knowledge about objects and their environment to help with a similar task? Even more interestingly, can we gather this information from language, using readily available resources such as language models trained on a large collection of unlabeled text? In this paper, we introduce a method called HOLM, …

Content truncated to 500 words