Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 8806–8811, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 8806 Towards Understanding Sample Variance in Visually Grounded Language Generation: Evaluations and Observations Wanrong Zhu†, Xin Eric Wang‡, Pradyumna Narayana*, Kazoo Sone*, Sugato Basu*, William Yang Wang† †UC Santa Barbara, ‡UC Santa Cruz, *Google {wanrongzhu,william}@cs.ucsb.edu, xwang366@ucsc.edu, {pradyn,sone,sugato}@google.com Abstract A major challenge in visually grounded lan- guage generation is to build robust benchmark datasets and models that can generalize well in real-world settings. To do this, it is crit- ical to ensure that our evaluation protocols are correct, and benchmarks are reliable. In this work, we set forth to design a set of ex- periments to understand an important but of- ten ignored problem in visually grounded lan- guage generation: given that humans have dif- ferent utilities and visual attention, how will the sample variance in multi-reference datasets affect the models’ performance? Empirically, we study several multi-reference datasets and corresponding vision-and-language tasks. We show that it is of paramount importance to report variance in experiments; that human- generated references could vary drastically in different datasets/tasks, revealing the nature of each task; that metric-wise, CIDEr has shown systematically larger variances than oth- ers. Our evaluations on reference-per-instance shed light on the design of reliable datasets in the future. 1 Introduction Natural Language Generation (NLG) is a chal- lenging problem in Natural Language Processing (NLP)—the complex nature of NLG tasks arise particularly in the output space. In contrast to text classification or regression problems with finite out- put space, generation could be seen as a combina- torial optimization problem, where we often have exponentially many options |V |ℓ (here |V | is the size of the vocabulary and ℓ is the sentence length). With the advances of both Computer Vision and NLP techniques in deep learning, there have been growing interests in visually grounded NLG tasks, such as image captioning (Hodosh et al., 2013; Young et al., 2014; Lin et al., 2014; Vedantam et al., 2015), video captioning (Xu et al., 2016; 1. This group of folks comprising runners and bikers, some wearing identifying numbers, look like they are getting ready for a marathon. 2. A runner in yellow has a convoy of motorcycles following behind him on a highway as bystanders watch. 3. Marathon runners are running down a street with motorcyclists nearby. 4. A runner in the middle of a race running along side the road. 5. A man in a yellow shirt is running in a race. 1. This group of folks comprising runners and bikers, some wearing identifying numbers, look like they are getting ready for a marathon. 2. A runner in yellow has a convoy of motorcycles following behind him on a highway as bystanders watch. 3. A man in a yellow shirt is running in a race. Figure 1: An image with three parallel captions from the Flickr30k dataset. Words in the same colors refer to the same objects. Wang et al., 2019; Chen and …

Content truncated to 500 words