Leveraging Past References for Robust Language Grounding

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 430–440 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 430 Leveraging Past References for Robust Language Grounding Subhro Roy∗, Michael Noseworthy∗, Rohan Paul, Daehyung Park, Nicholas Roy Computer Science and Artificial Intelligence Laboratory Massachusetts Institute of Technology {subhro, mnosew, rohanp, daehyung, nickroy}@csail.mit.edu Abstract Grounding referring expressions to objects in an environment has traditionally been consid- ered a one-off, ahistorical task. However, in realistic applications of grounding, multi- ple users will repeatedly refer to the same set of objects. As a result, past referring ex- pressions for objects can provide strong sig- nals for grounding subsequent referring ex- pressions. We therefore reframe the ground- ing problem from the perspective of corefer- ence detection and propose a neural network that detects when two expressions are referring to the same object. The network combines in- formation from vision and past referring ex- pressions to resolve which object is being re- ferred to. Our experiments show that detecting referring expression coreference is an effective way to ground objects described by subtle vi- sual properties, which standard visual ground- ing models have difficulty capturing. We also show the ability to detect object coreference allows the grounding model to perform well even when it encounters object categories not seen in the training data. 1 Introduction Grounding referring expressions to objects in an environment is a key Artificial Intelligence chal- lenge spanning computer vision and natural lan- guage processing. Past work in referring expres- sion grounding has focused on understanding the ways a human might resolve ambiguity that arises when multiple similar objects are in a scene – for example, by referring to visual properties or spatial relations (Nagaraja et al., 2016; Hu et al., 2017). However, most previous work treats it as an ahistorical task: a user is presented with an im- age and a referring expression and only features from the current referring expression are used to ∗Equal contribution. Utterances from a User: (1) The open text book. (2) Page with a dark border and images of two people at the bottom. (3) The open book with typed pages. (4) Pages of handwritten notes under the open book. Figure 1: An example where a single user repeatedly refers to objects in the same scene. The color identifies which bounding box is being referred to. determine which object is being referred to. How- ever, in many scenarios where a grounding system can be deployed, grounding is not an isolated one- off task. Instead, users will repeatedly refer to the same set of objects. In a household environment, a robot equipped with a grounding system will re- peatedly be asked to retrieve objects by the people who live there. Similarly, during a cooperative as- sembly task, a robot will repeatedly be asked for various parts or tools. Even in non-embodied sys- tems, such as conversational agents, a user will re- peatedly refer to different relevant entities. These scenarios present new challenges and op- …

Content truncated to 500 words