Structured Prediction with Output Embeddings for Semantic Image Annotation

Proceedings of NAACL-HLT 2016, pages 552–557, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Structured Prediction with Output Embeddings for Semantic Image Annotation Ariadna Quattoni1, Arnau Ramisa2, Pranava Swaroop Madhyastha3 Edgar Simo-Serra4, Francesc Moreno-Noguer2 1Xerox Research Center Europe, ariadna.quattoni@xrce.xerox.com 2 Institut de Rob`otica i Inform`atica Industrial (CSIC-UPC), {aramisa,fmoreno}@iri.upc.edu 3TALP Research Center, Universitat Polit`ecnica de Catalunya, pranava@cs.upc.edu 4Waseda University, esimo@aoni.waseda.jp Abstract We address the task of annotating images with semantic tuples. Solving this problem requires an algorithm able to deal with hundreds of classes for each argument of the tuple. In such contexts, data sparsity becomes a key challenge. We propose handling this spar- sity by incorporating feature representations of both the inputs (images) and outputs (ar- gument classes) into a factorized log-linear model. 1 Introduction Many important problems in machine learning can be framed as structured prediction tasks where the goal is to learn functions that map inputs to struc- tured outputs such as sequences, trees or general graphs. A wide range of applications involve learn- ing over large state spaces, e.g., if the output is a labeled graph, each node of the graph may take val- ues over a potentially large set of labels. Data spar- sity then becomes a challenge, as there will be many classes with very few training examples. Within this context, we are interested in the task of predicting semantic tuples for images. That is, given an input image we seek to predict what are the events or actions (referred here as predicates), who and what are the participants (referred here as actors) of the actions and where is the action taking place (referred here as locatives). For example, an image might be annotated with the semantic tuples: ⟨run, dog, park⟩ and ⟨play, dog, grass⟩. We call each field of a tuple an argument. To handle the data sparsity challenge imposed by the large state space, we will leverage an approach that has proven to be useful in multiclass and mul- tilabel prediction tasks (Weston et al., 2010; Akata et al., 2013). The idea is to represent a value for an argument a using a feature vector representation φ ∈ IRn. We will integrate this argument represen- tation into the structured prediction model. In summary, our main contribution is to pro- pose an approach that incorporates feature represen- tations of the outputs into a structured prediction model, and apply it to the problem of annotating images with semantic tuples. We present an experi- mental study using different output feature represen- tations and analyze how they affect performance for 2different argument types. Task:Semantic Tuple Image Annotation We will address the task of predicting se- mantic tuples for images. Following Farhadi et al. (2010), we will focus on a simple semantic repre- sentation that considers three basic arguments: pred- icate, actors and locatives. For example, in the tuple ⟨play, dog, grass⟩, “play” is the predicate, “dog” is the actor and “grass” is the locative. Given this representation, we can formally de- fine our problem …

Content truncated to 500 words