BERTHA: Video Captioning Evaluation Via Transfer-Learned Human Assessment

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1566–1575 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1566 BERTHA: Video Captioning Evaluation Via Transfer-Learned Human Assessment Luis Lebron1, Yvette Graham2, Kevin McGuinness1, Konstantinos Kouramas3, Noel E. O’Connor1 1Insight SFI Research Centre for Data Analytics, Dublin City University (DCU) 2School of Computer Science and Statistics, Trinity College Dublin 3Collins Aerospace luis.lebroncasas@insight-centre.org Abstract Evaluating video captioning systems is a challenging task with multiple challenges to consider. Firstly, the fluency of the caption, multiple actions taking place within a single scene, and estimation of what a human user might consider important in a video. Most metrics aim to measure how similar the system generated captions are to a single or a set of human-generated captions. This paper presents a new method based on a deep learning model to evaluate systems. The model is based on BERT language model, shown to work well across a range of NLP tasks. The aim is for the model to learn to perform an evaluation similar to that of a human. To do so, we use a dataset that contains human evaluation of system-generated captions. The dataset consists of human judgments of the quality of captions produced by the system participating in past TRECVid video to text tasks (Smeaton et al., 2006). These annotations will be made publicly available.The new video captioning evaluation metric, BERTHA, obtains favourable results, outperforming commonly applied metrics in some setups. Keywords: Video captioning, NLP, deep learning, learned metric 1. Introduction Automatic video captioning is a challenging multimodal task requiring the successful combination of computer vision and natural language generation. Systems aim to generate fluent natural language for videos of var- ious duration, ranging from generation of individual sentences that describe short videos (Awad et al., 2020a) to several sentences for longer videos (Tang et al., 2019; Caba Heilbron et al., 2015). This task has many challenges, with evaluation of the task itself being one of the most problematic. Video cap- tioning differs from other tasks by looking for accurate descriptions of specific objects and also requires accu- rate description of the context. A significant challenge in evaluating video captioning systems lies in the fact that there legitimately exists multiple and often very many distinct ways to describe what took place in a single video. The usual automatic metrics used for video captioning are borrowed from other tasks like machine translation or image captioning. Metrics like BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005), or CIDEr (Vedantam et al., 2015) are normally based on computing the overlap between the caption and the refer- ence sentence. However, they fail to include the context of the scene. New metrics like SPICE (Anderson et al., 2016) try to address this problem using a graph to represent the semantic content. Evaluating the accuracy of a given metric is ordinarily carried out by computing the correlation of its scores for systems with human assessment of the quality …

Content truncated to 500 words