Guiding Zero-Shot Paraphrase Generation with Fine-Grained Control Tokens

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 11–17 July 13-14, 2023 ©2023 Association for Computational Linguistics Leverage Points in Modality Shifts: Comparing Language-only and Multimodal Word Representations Aleksey Tikhonov Inworld.AI altsoph@gmail.com Lisa Bylinina∗ Center for Language and Cognition University of Groningen e.g.bylinina@rug.nl Denis Paperno∗ Institute for Language Sciences Utrecht University d.paperno@uu.nl Abstract Multimodal embeddings aim to enrich the se- mantic information in neural representations of language compared to text-only models. While different embeddings exhibit different applica- bility and performance on downstream tasks, little is known about the systematic representa- tion differences attributed to the visual modal- ity. Our paper compares word embeddings from three vision-and-language models (CLIP, OpenCLIP and Multilingual CLIP, Radford et al. 2021; Ilharco et al. 2021; Carlsson et al. 2022) and three text-only models, with static (FastText, Bojanowski et al., 2017) as well as contextual representations (multilingual BERT Devlin et al. 2018; XLM-RoBERTa, Conneau et al. 2019). This is the first large-scale study of the effect of visual grounding on language rep- resentations, including 46 semantic parameters. We identify meaning properties and relations that characterize words whose embeddings are most affected by the inclusion of visual modal- ity in the training data; that is, points where visual grounding turns out most important. We find that the effect of visual modality correlates most with denotational semantic properties re- lated to concreteness, but is also detected for several specific semantic classes, as well as for valence, a sentiment-related connotational property of linguistic expressions. 1 Introduction Linguistic representations developed by recent large pre-trained language models (LMs) (Devlin et al., 2018; Liu et al., 2019; Radford et al., 2019 a.o.) proved to be very useful across a variety of practical applications. This success has given a new life to the debate around extractability and quality of semantic information in representations trained solely on textual input. According to the ∗Equal contribution. widely supported argument, unless the textual data is grounded in a separate space (say, visual), the lin- guistic representations are bound to be semantically deficient (see Bender and Koller, 2020 a.o.). We aim to shed new empirical light on the dis- cussion of grounding in computational models by comparing language-only text representations to visually informed text representations. Recent advances produced empirically successful large models pre-trained on a combination of textual and visual data (Li et al., 2019; Tan and Bansal, 2019, 2020; Radford et al., 2021). While these multimodal systems have already given rise to a plethora of applications for language-and-vision (L&V) downstream tasks, there is still little work that directly compares textual representations of language-only models to those of multimodal ones (however, see Davis et al., 2019; Lüddecke et al., 2019; Pezzelle et al., 2021). In contrast to previous related work that focuses on model evaluation with respect to specific benchmarks, we look at the im- pact of visual grounding from a somewhat different, non-evaluation-based perspective. We do not aim to measure the representation quality with respect to some gold standard, but …

Content truncated to 500 words