Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 4685 - 4697 December 7-11, 2022 ©2022 Association for Computational Linguistics Context Matters for Image Descriptions for Accessibility: Challenges for Referenceless Evaluation Metrics Elisa Kreiss∗ Stanford University Cynthia Bennett Google Research Shayan Hooshmand Columbia University Eric Zelikman Stanford University Meredith Ringel Morris Google Research Christopher Potts Stanford University Abstract Few images on the Web receive alt-text descriptions that would make them accessible to blind and low vision (BLV) users. Image- based NLG systems have progressed to the point where they can begin to address this persistent societal problem, but these systems will not be fully successful unless we evaluate them on metrics that guide their development correctly. Here, we argue against current referenceless metrics – those that don’t rely on human-generated ground-truth descriptions – on the grounds that they do not align with the needs of BLV users. The fundamental shortcoming of these metrics is that they do not take context into account, whereas contextual information is highly valued by BLV users. To substantiate these claims, we present a study with BLV participants who rated descriptions along a variety of dimensions. An in-depth analysis reveals that the lack of context-awareness makes current referenceless metrics inadequate for advancing image accessibility. As a proof-of-concept, we provide a contextual version of the referenceless metric CLIPScore which begins to address the disconnect to the BLV data. An accessible HTML version of this paper is avail- able at https://elisakreiss.github.io/ contextual-description-evaluation/ paper/reflessmetrics.html 1 Introduction In the pursuit of ever more powerful image de- scription systems, we need evaluation metrics that provide a clear window into model capabilities. At present, we are seeing a rise in referenceless (or reference-free) metrics (Hessel et al., 2021; Lee et al., 2021a,b; Feinglass and Yang, 2021), build- ing on prior work in domains such as machine translation (Lo, 2019; Zhao et al., 2020) and sum- marization (Louis and Nenkova, 2013; Peyrard and ∗Corresponding author: ekreiss@stanford.edu Sculpture Gazebo Image description A freestanding, open, hexagonal gazebo with a dome-like roof in an idyllic park area. Good description Bad description Figure 1: Whether an image description makes an image accessible depends on the context in which the image ap- pears. Referenceless metrics like CLIPScore can’t cap- ture such context-sensitivity. We provide experimental evidence with blind and low vision (BLV) participants that this makes current referenceless metrics insufficient for evaluating image description quality. Gurevych, 2018; Gao et al., 2020; Deutsch et al., 2021). These metrics seek to estimate the quality of a text corresponding to an image without requiring ground-truth labels (i.e., reference descriptions), or crowd worker judgments. Thus, they offer the promise of quick and efficient evaluation of image description models, and are even suggested to be more reliable than existing reference-based met- rics (Kasai et al., 2022b,a). Here, we investigate the value of such metrics for a high social-impact domain: assessing the usefulness of image descrip- tions for blind and low vision (BLV) users. Automatically generating descriptions to make images …

Content truncated to 500 words