Visual Story Post-Editing

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6581–6586 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6581 Visual Story Post-Editing Ting-Yao Hsu1, Chieh-Yang Huang1, Yen-Chia Hsu2, Ting-Hao (Kenneth) Huang1 1Pennsylvania State University, State College, PA, USA 2Carnegie Mellon University, Pittsburgh, PA, USA 1{txh357, chiehyang, txh710}@psu.edu 2yenchiah@andrew.cmu.edu Abstract We introduce the first dataset for human edits of machine-generated visual stories and ex- plore how these collected edits may be used for the visual story post-editing task. The dataset, VIST-Edit1, includes 14,905 human- edited versions of 2,981 machine-generated visual stories. The stories were generated by two state-of-the-art visual storytelling models, each aligned to 5 human-edited versions. We establish baselines for the task, showing how a relatively small set of human edits can be leveraged to boost the performance of large visual storytelling models. We also discuss the weak correlation between automatic evalu- ation scores and human ratings, motivating the need for new automatic metrics. 1 Introduction Professional writers emphasize the importance of editing. Stephen King once put it this way: “to write is human, to edit is divine.” (King, 2000) Mark Twain had another quote: “Writing is easy. All you have to do is cross out the wrong words.” (Twain, 1876) Given that professionals re- vise and rewrite their drafts intensively, machines that generate stories may also benefit from a good editor. Per the evaluation of the first Visual Story- telling Challenge (Mitchell et al., 2018), the ability of an algorithm to tell a sound story is still far from that of a human. Users will inevitably need to edit generated stories before putting them to real uses, such as sharing on social media. We introduce the first dataset for human edits of machine-generated visual stories, VIST-Edit, and explore how these collected edits may be used for the task of visual story post-editing (see Fig- ure 1). The original visual storytelling (VIST) task, as introduced by Huang et al. (2016), takes 1VIST-Edit: https://github.com/tingyaohsu/VIST-Edit Figure 1: A machine-generated visual story (a) (by GLAC), its human-edited (b) and machine-edited (c) (by LSTM) version. a sequence of five photos as input and generates a short story describing the photo sequence. Huang et al. also released the VIST dataset, containing 20,211 photo sequences, aligned to human-written stories. On the other hand, the automatic post- editing task revises the story generated from vi- sual storytelling models, given both a machine- generated story and a photo sequence. Automatic post-editing treats the VIST system as a black box that is fixed and not modifiable. Its goal is to correct systematic errors of the VIST system and leverage the user edit data to improve story quality. In this paper, we (i) collect human edits for machine-generated stories from two different state-of-the-art models, (ii) analyze what people edited, and (iii) advance the task of visual story post-editing. In addition, we establish baselines for the task, and discuss the weak correlation be- tween automatic evaluation scores and human rat- ings, motivating …

Content truncated to 500 words