SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 2250–2260 August 1–6, 2021. ©2021 Association for Computational Linguistics 2250 SMURF: SeMantic and linguistic UndeRstanding Fusion for Caption Evaluation via Typicality Analysis Joshua Feinglass and Yezhou Yang Arizona State University {joshua.feinglass,yz.yang}@asu.edu Abstract The open-ended nature of visual captioning makes it a challenging area for evaluation. The majority of proposed models rely on special- ized training to improve human-correlation, re- sulting in limited adoption, generalizability, and explainabilty. We introduce “typicality”, a new formulation of evaluation rooted in in- formation theory, which is uniquely suited for problems lacking a definite ground truth. Typ- icality serves as our framework to develop a novel semantic comparison, SPARCS, as well as referenceless fluency evaluation metrics. Over the course of our analysis, two separate dimensions of fluency naturally emerge: style, captured by metric SPURTS, and grammar, captured in the form of grammatical outlier penalties. Through extensive experiments and ablation studies on benchmark datasets, we show how these decomposed dimensions of semantics and fluency provide greater system- level insight into captioner differences. Our proposed metrics along with their combina- tion, SMURF, achieve state-of-the-art corre- lation with human judgment when compared with other rule-based evaluation metrics1. 1 Introduction Visual captioning serves as a foundation for im- age/video understanding tools and relies on cap- tion evaluation for identifying promising research directions. Rule-based caption evaluation ap- proaches like the n-gram based CIDEr (Vedantam et al., 2015) and parsed semantic proposal based SPICE (Anderson et al., 2016) specifically are able to provide researchers with meaningful feedback on what their algorithm is lacking. However, n- gram based methods are sensitive to stop words and sentence parsers are often inconsistent, leading to Liu et al. (2017) showing that neither method 1SMURF source codes and data will be released at https: //github.com/JoshuaFeinglass/SMURF. Figure 1: Scatter plot utilizing standardizations of SPARCS and SPURTS. The ground truth captions are sourced from the Karpathy test split of the COCO dataset (Chen et al., 2015; Karpathy and Fei-Fei, 2015) with one used as a baseline for automatic caption- ers (Cornia et al., 2020; Pan et al., 2020; Vinyals et al., 2015). For each captioner, a 75% confidence ellipse (1.15 standard deviations from the mean) is generated. A caption near the centroid of each captioner is shown as an example along with the caption scores from 100 randomly sampled images. The normalized ellipse overlap between an automatic captioner and human captions, H∩M Area M Area , gives an overall evaluation of typ- ical performance at a system-level on a scale of 0 to 1, with 1 being human-caption level. fully captures either the fluency or the semantic meaning of text. More recently proposed metrics attempt to learn cues of caption quality by training models via image grounding techniques (Cui et al., 2018) or human and generated captions (Sellam et al., 2020). These approaches, however, lack gen- erality, require domain specific training, and offer little insight for …

Content truncated to 500 words