Generating Expressions that Refer to Visible Objects

Proceedings of NAACL-HLT 2013, pages 1174–1184, Atlanta, Georgia, 9–14 June 2013. c⃝2013 Association for Computational Linguistics Generating Expressions that Refer to Visible Objects Margaret Mitchell Johns Hopkins HLTCOE m.mitchell@jhu.edu Kees van Deemter University of Aberdeen k.vdeemter@abdn.ac.uk Ehud Reiter University of Aberdeen e.reiter@abdn.ac.uk Abstract We introduce a novel algorithm for generat- ing referring expressions, informed by human and computer vision and designed to refer to visible objects. Our method separates abso- lute properties like color from relative proper- ties like size to stochastically generate a di- verse set of outputs. Expressions generated using this method are often overspecified and may be underspecified, akin to expressions produced by people. We call such expressions identifying descriptions. The algorithm out- performs the well-known Incremental Algo- rithm (Dale and Reiter, 1995) and the Graph- Based Algorithm (Krahmer et al., 2003; Vi- ethen et al., 2008) across a variety of images in two domains. We additionally motivate an evaluation method for referring expression generation that takes the proposed algorithm’s non-determinism into account. 1 Introduction Referring expression generation (REG) is the task of generating an expression that can identify a ref- erent to a listener. These expressions generally take the form of a definite noun phrase such as “the large orange plate” or “the furry running dog”. Research in REG primarily focuses on the subtask of select- ing a set of properties that may be used to construct the final surface expression, e.g., ⟨color:orange, size:large, type:plate⟩. This property selection task is optimized to meet different goals: for example, to be identical to those a person would generate in the same situation, or to be unique to the intended referent and no other item in the discourse. We focus on the task of generating referring ex- pressions for visible objects, specifically with the goal of generating descriptive, human-like referring expressions. We are motivated by the desire to con- nect this algorithm to input from a computer vision system, and discuss how this may work through- out the paper. Computer vision (CV) does not yet reliably provide features for some of the most fre- quent properties that people use in visual descrip- tion (in particular, size-based features), and so we use a gold-standard visual input, evaluating purely on REG. The proposed algorithm, which we call the Visible Objects Algorithm, is designed to ap- proximate human variation identifying an object in a group of visible, real world objects. Our primary contributions are the following. Background for each issue is provided in Section 2: 1. An approach accounting for overspecification, underspecification, and some of the known ef- fects of vision on reference. 2. A function to approximate the stochastic nature of reference. This reflects that people will pro- duce different references to the same object. 3. A separation between absolute properties like color, which may be detected directly by CV, from relative properties like size and loca- tion, which require reasoning over visual fea- tures to determine an appropriate form (e.g., height/width and distance features between pixels are available from a …

Content truncated to 500 words