PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2040–2050 August 1–6, 2021. ©2021 Association for Computational Linguistics 2040 PIGLeT: Language Grounding Through Neuro-Symbolic Interaction in a 3D World Rowan Zellers♠ Ari Holtzman♠ Matthew Peters~ Roozbeh Mottaghi~ Aniruddha Kembhavi~ Ali Farhadi♠ Yejin Choi♠~ ♠Paul G. Allen School of Computer Science & Engineering, University of Washington ~Allen Institute for Artificial Intelligence https://rowanzellers.com/piglet Abstract We propose PIGLeT: a model that learns phys- ical commonsense knowledge through interac- tion, and then uses this knowledge to ground language. We factorize PIGLeT into a physi- cal dynamics model, and a separate language model. Our dynamics model learns not just what objects are but also what they do: glass cups break when thrown, plastic ones don’t. We then use it as the interface to our language model, giving us a unified model of linguis- tic form and grounded meaning. PIGLeT can read a sentence, simulate neurally what might happen next, and then communicate that re- sult through a literal symbolic representation, or natural language. Experimental results show that our model ef- fectively learns world dynamics, along with how to communicate them. It is able to cor- rectly forecast “what happens next” given an English sentence over 80% of the time, outper- forming a 100x larger, text-to-text approach by over 10%. Likewise, its natural language sum- maries of physical interactions are also judged by humans as more accurate than LM alter- natives. We present comprehensive analysis showing room for future work. 1 Introduction As humans, our use of language is linked to the physical world. To process a sentence like “the robot turns on the stove, with a pan on it” (Figure 1) we might imagine a physical Pan object. This meaning representation in our heads can be seen as a part of our commonsense world knowledge, about what a Pan is and does. We might reasonably predict that the Pan will become Hot – and if there’s an Egg on it, it would become cooked. As humans, we learn such a commonsense world model through interaction. Young children learn to reason physically about basic objects by manip- ulating them: observing the properties they have, Language Model PIGLeT t t+1 The robot turns on the stove, with a pan on it. isBroken: True isCooked: False Temperature: RoomTemp Name: Egg ... isBroken: True isCooked: True Temperature: Hot Name: Egg ... <heatUp, Pan> Physical Dynamics Model The pan is now hot and the egg becomes cooked. Figure 1: PIGLeT. Through physical interaction in a 3D world, we learn a model for what actions do to objects. We use our physical model as an interface for a lan- guage model, jointly modeling elements of language form and meaning. Given an action expressed symbol- ically or in English, PIGLeT can simulate what might happen next, expressing it symbolically or in English. and how they change if an action is applied on them (Smith and Gasser, 2005). This …

Content truncated to 500 words