Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training

Proceedings of the 25th Conference on Computational Natural Language Learning (CoNLL), pages 158–170 November 10–11, 2021. ©2021 Association for Computational Linguistics 158 Learning Zero-Shot Multifaceted Visually Grounded Word Embeddings via Multi-Task Training Hassan Shahmohammadi Hendrik P. A. Lensch R. Harald Baayen University of Tübingen hassan.shahmohammadi@uni-tuebingen.de {hendrik.lensch, harald.baayen}@uni-tuebingen.de Abstract Language grounding aims at linking the sym- bolic representation of language (e.g., words) into the rich perceptual knowledge of the out- side world. The general approach is to embed both textual and visual information into a com- mon space -the grounded space- confined by an explicit relationship. We argue that since concrete and abstract words are processed dif- ferently in the brain, such approaches sacrifice the abstract knowledge obtained from textual statistics in the process of acquiring percep- tual information. The focus of this paper is to solve this issue by implicitly grounding the word embeddings. Rather than learning two mappings into a joint space, our approach inte- grates modalities by implicit alignment. This is achieved by learning a reversible mapping between the textual and the grounded space by means of multi-task training. Intrinsic and extrinsic evaluations show that our way of vi- sual grounding is highly beneficial for both ab- stract and concrete words. Our embeddings are correlated with human judgments and out- perform previous works using pretrained word embeddings on a wide range of benchmarks. Our grounded embeddings are publicly avail- able here. 1 Introduction The distributional hypothesis asserts that words oc- curring in similar contexts are semantically related (Harris, 1954). Current state-of-the-art word em- bedding models (Pennington et al., 2014; Peters et al., 2018a), despite their successful application to various NLP tasks (Wang et al., 2018), suffer from the lack of grounding in general knowledge (Harnad, 1990; Burgess, 2000), such as captured by human perceptual and motor systems (Pulver- müller, 2005; Therriault et al., 2009). To overcome this limitation, research has been directed to link- ing word embeddings to perceptual knowledge in visual scenes. Most studies have attempted to bring visual and language representations into close vicin- ity in a common feature space (Silberer and Lapata, 2014; Kurach et al., 2017; Kiela et al., 2018). However, studies of human cognition indicate that the brain processes abstract and concrete words differently (Paivio, 1990; Anderson et al., 2017) due to the difference in associated sensory per- ception. According to Montefinese (2019), sim- ilar activity for both categories are observed in the perirhinal cortext, a region related to memory and recognition, whereas in the parahippocampal cortex, associated with memory formation, higher activity only occurs for abstract words. We argue that forcing the textual and visual modalities to be represented in a shared space causes grounded embeddings to suffer from the bias towards concrete words as reported by Park and Myaeng (2017); Kiela et al. (2018). Therefore, we propose a zero-shot approach that implicitly in- tegrates perceptual knowledge into pre-trained tex- tual embeddings (GloVe (Pennington et al., 2014) and fastText (Bojanowski et al., 2017)) via multi- task training. Our approach learns multifaceted grounded …

Content truncated to 500 words