Learning Concept Taxonomies from Multi-modal Data

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1791–1801, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Learning Concept Taxonomies from Multi-modal Data Hao Zhang1, Zhiting Hu1, Yuntian Deng1, Mrinmaya Sachan1, Zhicheng Yan2, Eric P. Xing1 1Carnegie Mellon University, 2UIUC {hao,zhitingh,yuntiand,mrinmays,epxing}@cs.cmu.edu Abstract We study the problem of automatically building hypernym taxonomies from tex- tual and visual data. Previous works in taxonomy induction generally ignore the increasingly prominent visual data, which encode important perceptual semantics. Instead, we propose a probabilistic model for taxonomy induction by jointly leverag- ing text and images. To avoid hand-crafted feature engineering, we design end-to-end features based on distributed representa- tions of images and words. The model is discriminatively trained given a small set of existing ontologies and is capable of building full taxonomies from scratch for a collection of unseen conceptual label items with associated images. We evalu- ate our model and features on the WordNet hierarchies, where our system outperforms previous approaches by a large gap. 1 Introduction Human knowledge is naturally organized as se- mantic hierarchies. For example, in WordNet (Miller, 1995), specific concepts are categorized and assigned to more general ones, leading to a semantic hierarchical structure (a.k.a taxonomy). A variety of NLP tasks, such as question answer- ing (Harabagiu et al., 2003), document cluster- ing (Hotho et al., 2002) and text generation (Biran and McKeown, 2013) can benefit from the con- ceptual relationship present in these hierarchies. Traditional methods of manually constructing taxonomies by experts (e.g. WordNet) and interest communities (e.g. Wikipedia) are either knowl- edge or time intensive, and the results have lim- ited coverage. Therefore, automatic induction of taxonomies is drawing increasing attention in both (a) Input Seafish Shark Ray Seafish Ray Shark “seafish, such as sharks and rays…” “shark and ray are a group of seafish…” “either ray or shark lives in …” (b) Output visual similarity wordvec closeness Figure 1: An overview of our system. (a) Input: a collection of label items, represented by text and images; (b) Output: we build a taxonomy from scratch by extracting features based on distributed representations of text and images. NLP and computer vision. On one hand, a num- ber of methods have been developed to build hi- erarchies based on lexical patterns in text (Yang and Callan, 2009; Snow et al., 2006; Kozareva and Hovy, 2010; Navigli et al., 2011; Fu et al., 2014; Bansal et al., 2014; Tuan et al., 2015). These works generally ignore the rich visual data which encode important perceptual semantics (Bruni et al., 2014) and have proven to be complemen- tary to linguistic information and helpful for many tasks (Silberer and Lapata, 2014; Kiela and Bot- tou, 2014; Zhang et al., 2015; Chen et al., 2013). On the other hand, researchers have built visual hi- erarchies by utilizing only visual features (Griffin and Perona, 2008; Yan et al., 2015; Sivic et al., 2008). The resulting hierarchies are limited in in- terpretability and usability for knowledge transfer. Hence, we propose …

Content truncated to 500 words