Towards Explainable Evaluation of Language Models on the Semantic Similarity of Visual Concepts

Proceedings of the 29th International Conference on Computational Linguistics, pages 3639–3658 October 12–17, 2022. 3639 Towards explainable evaluation of language models on the semantic similarity of visual concepts Maria Lymperaiou, George Manoliadis, Orfeas Menis Mastromichalakis, Edmund G. Dervakos and Giorgos Stamou Artificial Intelligence and Learning Systems Laboratory School of Electrical and Computer Engineering National Technical University of Athens {marialymp, eddiedervakos}@islab.ntua.gr, gmanoliad@mail.ntua.gr, menorf@ails.ece.ntua.gr, gstam@cs.ntua.gr Abstract Recent breakthroughs in NLP research, such as the advent of Transformer models have indis- putably contributed to major advancements in several tasks. However, few works research ro- bustness and explainability issues of their eval- uation strategies. In this work, we examine the behavior of high-performing pre-trained language models, focusing on the task of se- mantic similarity for visual vocabularies. First, we address the need for explainable evaluation metrics, necessary for understanding the con- ceptual quality of retrieved instances. Our pro- posed metrics provide valuable insights in local and global level, showcasing the inabilities of widely used approaches. Secondly, adversarial interventions on salient query semantics expose vulnerabilities of opaque metrics and highlight patterns in learned linguistic representations. 1 Introduction Semantic similarity between pairs of sentences serves a large variety of applications in the field of natural language processing, such as document re- trieval, text classification, question answering and others. Even though such tasks have risen in pop- ularity since the introduction of the Transformers (Vaswani et al., 2017), and despite the attention given on robustness and transparency of NLP trans- formers (Hendrycks et al., 2020; Hsieh et al., 2019; Baan et al., 2019) few efforts have addressed ex- plainable evaluation (Leiter et al., 2022). Text-Image retrieval is a real world semantic similarity application where the task is to feed a textual input to a system, and receive an image as a response. Visual details of the retrieved in- stance need to accurately correspond to the textual descriptions, often in a fine-grained fashion. Any mismatch between modalities can be easily per- ceived by humans, and captured by automated met- rics. Such evident disagreements can act as starting points for further investigation, revealing inner pro- cesses on the semantic matching procedures. In this work, we aim to unveil the evaluation strategy of semantic similarity models. Specifi- cally, we apply pre-trained transformers on visual vocabularies and obtain results via ranking. First, we address the shortcomings of traditional ranking metrics (Manning et al., 2008), which provide ei- ther a binary answer (item found in top-k items or not), or position-informed variants (item found in the k-th position). However, such measures cannot provide detailed insights regarding the contribu- tion of the scene constituents to the rank position. For example, if an instance is ranked in the k-th position, items in previous k-1 positions may be highly relevant to the ground truth one or on the contrary, highly irrelevant. To this end, we propose novel explainable ranking evaluation metrics that decompose and quantify the conceptual differences between ground truth and retrieved instances in lo- cal and global level. Even then, we observe that …

Content truncated to 500 words