Learning to Rank Visual Stories From Human Ranking Data

Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 6365 - 6378 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Learning to Rank Visual Stories from Human Ranking Data Chi-Yang Hsu1∗, Yun-Wei Chu2∗, Vincent Chen3∗, Kuan-Chieh Lo3, Chacha Chen4, Ting-Hao (Kenneth) Huang1, Lun-Wei Ku3 Pennsylvania State University 1, Purdue University 2, Institute of Information Science, Academia Sinica3, University of Chicago4 {cxh5437, txh710}@psu.edu, {chu198}@purdue.edu, {vincent0110, kclo7898, lwku}@iis.sinica.edu.tw, {chacha}@uchicago.edu Abstract Visual storytelling (VIST) is a typical vision and language task that has seen extensive de- velopment in the natural language generation research domain. However, it remains unclear whether conventional automatic evaluation met- rics for text generation are applicable on VIST. In this paper, we present the VHED (VIST Human Evaluation Data) dataset, which first re-purposes human evaluation results for au- tomatic evaluation; hence we develop Vrank (VIST ranker), a novel reference-free VIST metric for story evaluation.1 We first show that the results from commonly adopted automatic metrics for text generation have little correla- tion with those obtained from human evalua- tion, which motivates us to directly utilize hu- man evaluation results to learn the automatic evaluation model. In the experiments, we evalu- ate the generated texts to predict story ranks us- ing our model as well as other reference-based and reference-free metrics. Results show that Vrank prediction is significantly more aligned to human evaluation than other metrics with al- most 30% higher accuracy when ranking story pairs. Moreover, we demonstrate that only Vrank shows human-like behavior in its strong ability to find better stories when the quality gap between two stories is high. Finally, we show the superiority of Vrank by its general- izability to pure textual stories, and conclude that this reuse of human evaluation results puts Vrank in a strong position for continued future advances. 1 Introduction In visual storytelling (VIST) (Huang et al., 2016), a generation model tells a short story to describe the given five images. Automatic generation of visual stories is challenging because it has the complex- ity of cross-modal understanding with the diversity ∗* denotes equal contribution 1Dataset VHED and metric Vrank can be found on GitHub: https://github.com/AcademiaSinicaNLPLab/ VHED.git the city was very busy. there were many different kinds of bikes. some were very unique. they were all very fast. i had a great time. i went to the park station. it was a train trip to the museum. the train was very long. we had to go on our way out of the trains. this dog is so happy to see us. Reference: i decided my dog would like a train ride. off to the train station we go. this is the train we will be taking our short trip on. my friend is the conductor. he is getting ready to attach the cars. here is the train all together. as you can see, my dog had a fantastic time. Model 1(BLEU-1: 0.605, Human Rankers: ) Model 2(BLEU-1: 0.354, Human Rankers: ) Figure 1: Ranking of two …

Content truncated to 500 words