Revisiting Document Representations for Large-Scale Zero-Shot Learning

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3117–3128 June 6–11, 2021. ©2021 Association for Computational Linguistics 3117 Revisiting Document Representations for Large-Scale Zero-Shot Learning Jihyung Kil The Ohio State University Columbus, Ohio, USA kil.5@osu.edu Wei-Lun Chao The Ohio State University Columbus, Ohio, USA chao.209@osu.edu Abstract Zero-shot learning aims to recognize unseen objects using their semantic representations. Most existing works use visual attributes la- beled by humans, not suitable for large-scale applications. In this paper, we revisit the use of documents as semantic representations. We ar- gue that documents like Wikipedia pages con- tain rich visual information, which however can easily be buried by the vast amount of non-visual sentences. To address this issue, we propose a semi-automatic mechanism for visual sentence extraction that leverages the document section headers and the clustering structure of visual sentences. The extracted vi- sual sentences, after a novel weighting scheme to distinguish similar classes, essentially form semantic representations like visual attributes but need much less human effort. On the Ima- geNet dataset with over 10,000 unseen classes, our representations lead to a 64% relative im- provement against the commonly used ones. 1 Introduction Algorithms for visual recognition usually require hundreds of labeled images to learn how to classify an object (He et al., 2016). In reality, however, the frequency of observing an object follows a long- tailed distribution (Zhu et al., 2014): many objects do not appear frequently enough for us to collect sufficient images. Zero-shot learning (ZSL) (Lam- pert et al., 2009), which aims to build classifiers for unseen object classes using their semantic represen- tations, has thus emerged as a promising paradigm for recognizing a large number of classes. Being the only information of unseen objects, how well the semantic representations describe the visual appearances plays a crucial role in ZSL. One popular choice is visual attributes (Lampert et al., 2009; Patterson and Hays, 2012; Wah et al., 2011) carefully annotated by humans. For example, the bird “Red bellied Woodpecker” has the “capped head pattern” and “pointed wing shape”. While Figure 1: An illustration of our ZSL approach, which recog- nizes the input image by comparing it to the visual sentences of documents. Here we show two documents, one for “Tiger” and one for “Lion”. The gray area highlights the extracted visual sentences (red: by section headers; blue: by clustering). strictly tied to visual appearances, visual attributes are laborious to collect, limiting their applicability to small-scale problems with hundreds of classes. For large-scale problems like ImageNet (Deng et al., 2009) that has more than 20, 000 classes, ex- isting ZSL algorithms (Frome et al., 2013; Norouzi et al., 2013) mostly resort to word vectors of classes names (Mikolov et al., 2013; Pennington et al., 2014) that are automatically extracted from large corpora like Common Crawl. While almost labor free, word vectors are purely text-driven and barely aligned with visual information. As a result, the state-of-the-art ZSL accuracy on ImageNet falls …

Content truncated to 500 words