Know What You Don’t Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 654–659 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 654 Know What You Don’t Know: Modeling a Pragmatic Speaker that Refers to Objects of Unknown Categories Sina Zarrieß Faculty of Linguistics and Literary Studies Bielefeld University, Germany sina.zarriess@uni-bielefeld.de David Schlangen Linguistics Department University of Potsdam, Germany david.schlangen@uni-potsdam.de Abstract Zero-shot learning in Language & Vision is the task of correctly labelling (or naming) objects of novel categories. Another strand of work in L&V aims at pragmatically informative rather than “correct” object descriptions, e.g. in ref- erence games. We combine these lines of re- search and model zero-shot reference games, where a speaker needs to successfully refer to a novel object in an image. Inspired by models of “rational speech acts”, we extend a neural generator to become a pragmatic speaker rea- soning about uncertain object categories. As a result of this reasoning, the generator pro- duces fewer nouns and names of distractor cat- egories as compared to a literal speaker. We show that this conversational strategy for deal- ing with novel objects often improves commu- nicative success, in terms of resolution accu- racy of an automatic listener. 1 Introduction It is commonly agreed that even massive resources for language & vision (Deng et al., 2009; Chen et al., 2015; Krishna et al., 2017) will never fully cover the huge range of objects to be found “in the wild”. This motivates research in zero-shot learning (Lampert et al., 2009; Socher et al., 2013; Hendricks et al., 2016), which aims at predicting correct labels or names for objects of novel cat- egories, typically via external lexical knowledge such as, e.g., word embeddings. More generally, however, uncertain knowledge of the world that surrounds us, including novel ob- jects, is not only a machine learning challenge: it is simply a very common aspect of human com- munication, as speakers rarely have perfect rep- resentations of their environment. Precisely the richness of verbal interaction allows us to com- municate these uncertainties and to collaborate to- wards communicative success (Clark and Wilkes- Gibbs, 1986). Figure 1 illustrates this general refexp: right thingy refexp: left blue Figure 1: RefCOCO expressions referring to diffi- cult/unknown objects point with two examples from the RefCOCO cor- pus (Yu et al., 2016), providing descriptions of vi- sual objects from an interactive reference game. Here, the use of the unspecific thingy and the omission of a noun in left blue can be seen as prag- matically plausible strategies that avoid confusing the listener with potentially inaccurate names for difficult-to-name objects. While there has been a lot of recent and traditional research on pragmat- ically informative object descriptions in reference games (Mao et al., 2016; Yu et al., 2017; Cohn- Gordon et al., 2018; Dale and Reiter, 1995; Frank and Goodman, 2012), conversational strategies for dealing with uncertainties like novel categories are largely understudied in computational pragmatics, though see, e.g., work by Fang et al. (2014). …

Content truncated to 500 words