Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation

Word Representation Learning in Multimodal Pre-Trained Transformers: An Intrinsic Evaluation Sandro Pezzelle, Ece Takmaz, Raquel Fern´andez Institute for Logic, Language and Computation University of Amsterdam, The Netherlands {s.pezzelle|e.takmaz|raquel.fernandez}@uva.nl Abstract This study carries out a systematic intrin- sic evaluation of the semantic representations learned by state-of-the-art pre-trained multi- modal Transformers. These representations are claimed to be task-agnostic and shown to help on many downstream language-and-vision tasks. However, the extent to which they align with human semantic intuitions remains un- clear. We experiment with various models and obtain static word representations from the contextualized ones they learn. We then eval- uate them against the semantic judgments pro- vided by human speakers. In line with previous evidence, we observe a generalized advantage of multimodal representations over language- only ones on concrete word pairs, but not on abstract ones. On the one hand, this confirms the effectiveness of these models to align lan- guage and vision, which results in better se- mantic representations for concepts that are grounded in images. On the other hand, mod- els are shown to follow different represen- tation learning patterns, which sheds some light on how and when they perform multi- modal integration. 1 Introduction Increasing evidence indicates that the meaning of words is multimodal: Human concepts are grounded in our senses (Barsalou, 2008; De Vega et al., 2012), and the sensory-motor experiences humans have with the world play an important role in determining word meaning (Meteyard et al., 2012). Since (at least) the first operationalizations of the distributional hypothesis, however, standard NLP approaches to derive meaning representa- tions of words have solely relied on information extracted from large text corpora, based on the generalized assumption that the meaning of a word can be inferred from the effects it has on its linguistic context (Harris, 1954; Firth, 1957). Language-only semantic representations, from pi- oneering ‘count’ vectors (Landauer and Dumais, 1997; Turney and Pantel, 2010; Pennington et al., 2014) to either static (Mikolov et al., 2013) or contextualized (Peters et al., 2018; Devlin et al., 2019) neural network-based embeddings, have proven extremely effective in many lin- guistic tasks and applications, for which they constantly increased state-of-the-art performance. However, they naturally have no connection with the real-world referents they denote (Baroni, 2016). As such, they suffer from the symbol grounding problem (Harnad, 1990), which in turn limits their cognitive plausibility (Rotaru and Vigliocco, 2020). To overcome this limitation, several methods have been proposed to equip language-only rep- resentations with information from concurrent modalities, particularly vision. Until not long ago, the standard approach aimed to leverage the com- plementary information conveyed by language and vision—for example, that bananas are yellow (vi- sion) and rich in potassium (language)—by build- ing richer multimodal representations (Beinborn et al., 2018). Overall, these representations have proved advantageous over purely textual ones in a wide range of tasks and evaluations, including the approximation of human semantic similarity/ relatedness judgments provided by benchmarks like SimLex999 (Hill et al., 2015) or MEN (Bruni et al., 2014). This was taken as …

Content truncated to 500 words