ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 569–579 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 569 ZEST: Zero-shot Learning from Text Descriptions using Textual Similarity and Visual Summarization Tzuf Paz-Argaman1, Yuval Atzmon2, Gal Chechik1,2, and Reut Tsarfaty1,3 1Bar-Ilan University 2NVIDIA Research 3Allen Institute for Artificial Intelligence {tzufar,reut.tsarfaty}@gmail.com Abstract We study the problem of recognizing visual entities from the textual descriptions of their classes. Specifically, given birds’ images with free-text descriptions of their species, we learn to classify images of previously-unseen species based on specie descriptions. This setup has been studied in the vision commu- nity under the name zero-shot learning from text, focusing on learning to transfer knowl- edge about visual aspects of birds from seen classes to previously-unseen ones. Here, we suggest focusing on the textual description and distilling from the description the most rele- vant information to effectively match visual features to the parts of the text that discuss them. Specifically, (1) we propose to lever- age the similarity between species, reflected in the similarity between text descriptions of the species. (2) we derive visual summaries of the texts, i.e., extractive summaries that focus on the visual features that tend to be reflected in images. We propose a simple attention-based model augmented with the similarity and vi- sual summaries components. Our empirical re- sults consistently and significantly outperform the state-of-the-art on the largest benchmarks for text-based zero-shot learning, illustrating the critical importance of texts for zero-shot image-recognition. 1 Introduction In computer vision, zero shot-learning (ZSL) for image classification is the problem of classifying images given auxiliary information. An image clas- sification model is trained to classify images from a pre-defined set of classes. At test time, images from new classes are given, and the task is to trans- fer knowledge learned from seen classes during training to unseen test classes. Figure 1: An illustration of textual similarity and visu- ally relevant descriptions in Wikipedia articles: (1) we aim to leverage the similarity within texts (red) via doc- ument clustering (bottom box); (2) we aim to extract similar (red) and dissimilar (black) visual descriptions, and remove non-visually relevant (blue) one. A common setup for ZSL assumes that the aux- iliary information is a set of semantically mean- ingful properties (called attributes) describing the class (e.g., black-beak, long-tail) (Wah et al., 2011; Farhadi et al., 2009). A different ZSL setup uses im- age captions as auxiliary information (Reed et al., 2016; Felix et al., 2018). Typically, this auxiliary information is manually collected by human raters for each image (test and train alike) and averaged across images. A more realistic approach relies on available online text descriptions of classes (e.g., Wikipedia) (Elhoseiny et al., 2017). It avoids ex- 570 pensive annotation and exposure to test images. In this work, we classify bird species according to Wikipedia descriptions. This task raises many challenges: (1) Differences between the birds are very small, which makes it a fine-grained classifi- cation task; (2) This is an expert task, and …

Content truncated to 500 words