Interactive Symbol Grounding with Complex Referential Expressions

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4863 - 4874 July 10-15, 2022 ©2022 Association for Computational Linguistics Interactive Symbol Grounding with Complex Referential Expressions Rimvydas Rubavicius and Alex Lascarides School of Informatics, University of Edinburgh 10 Crichton Street, Edinburgh EH8 9AB, UK rimvydas.rubavicius@ed.ac.uk, alex@inf.ed.ac.uk Abstract We present a procedure for learning to ground symbols from a sequence of stimuli consisting of an arbitrarily complex noun phrase (e.g. “all but one green square above both red circles.”) and its designation in the visual scene. Our distinctive approach combines: a) lazy few- shot learning to relate open-class words like green and above to their visual percepts; and b) symbolic reasoning with closed-class word categories like quantifiers and negation. We use this combination to estimate new train- ing examples for grounding symbols that oc- cur within a noun phrase but aren’t designated by that noun phase (e.g, red in the above ex- ample), thereby potentially gaining data effi- ciency. We evaluate the approach in a visual reference resolution task, in which the learner starts out unaware of concepts that are part of the domain model and how they relate to visual percepts. 1 Introduction The subfield of robotics known as Interactive Task Learning (ITL, see Laird et al. (2017) for a survey) addresses scenarios where a robot must learn to adapt its behaviour to novel and unforeseen ob- jects, relations, and attributes that are introduced into the environment after deployment. The ITL agent learns its novel task via evidence from its own actions and reactive guidance from a teacher. This paper focuses on symbol grounding (Harnad, 1999) in the context of ITL (Matuszek, 2018): the learner must use the teacher’s embodied natural language utterance and its context to learn a map- ping from natural language expressions to their denotations, given the visual percepts. There are two challenges in learning symbol grounding models (grounders) in ITL. Firstly, in contrast to many grounders (Ye et al., 2019; Datta et al., 2019), ITL requires incremental learning: knowledge is acquired piecemeal via an extended interaction, and it must influence planning as and when it occurs. Secondly, previous work limits the teacher’s language to bare nouns (e.g., square) or very short phrases (e.g., blue square, square above circle) (Hristov et al., 2018, 2019). But there’s evidence from Dale and Reiter (1995) that speak- ers use complex referring expressions even when simpler ones would successfully refer. Such lan- guage creates the possibility that novel symbols— neologisms—are introduced in a context where their denotation is not designated by the teacher. In this work we study the natural language of complex referential expressions (REs) like “a blue square be- hind both red circles” which teachers can use when designating an object. Our aim is for the learner to extract knowledge that improves their domain representation and state estimates—a necessary condition for successful planning. Contemporary grounders miss learning opportunities that complex REs afford: the RE ex- ample above …

Content truncated to 500 words