Reinforced Video Captioning with Entailment Rewards

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 979–985 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Reinforced Video Captioning with Entailment Rewards Ramakanth Pasunuru and Mohit Bansal UNC Chapel Hill {ram, mbansal}@cs.unc.edu Abstract Sequence-to-sequence models have shown promising improvements on the temporal task of video captioning, but they opti- mize word-level cross-entropy loss dur- ing training. First, using policy gra- dient and mixed-loss methods for re- inforcement learning, we directly opti- mize sentence-level task-based metrics (as rewards), achieving significant improve- ments over the baseline, based on both automatic metrics and human evaluation on multiple datasets. Next, we pro- pose a novel entailment-enhanced reward (CIDEnt) that corrects phrase-matching based metrics (such as CIDEr) to only al- low for logically-implied partial matches and avoid contradictions, achieving fur- ther significant improvements over the CIDEr-reward model. Overall, our CIDEnt-reward model achieves the new state-of-the-art on the MSR-VTT dataset. 1 Introduction The task of video captioning (Fig. 1) is an im- portant next step to image captioning, with ad- ditional modeling of temporal knowledge and action sequences, and has several applications in online content search, assisting the visually- impaired, etc. Advancements in neural sequence- to-sequence learning has shown promising im- provements on this task, based on encoder- decoder, attention, and hierarchical models (Venu- gopalan et al., 2015a; Pan et al., 2016a). How- ever, these models are still trained using a word- level cross-entropy loss, which does not correlate well with the sentence-level metrics that the task is finally evaluated on (e.g., CIDEr, BLEU). More- over, these models suffer from exposure bias (Ran- Figure 1: A correctly-predicted video caption gen- erated by our CIDEnt-reward model. zato et al., 2016), which occurs when a model is only exposed to the training data distribu- tion, instead of its own predictions. First, us- ing a sequence-level training, policy gradient ap- proach (Ranzato et al., 2016), we allow video captioning models to directly optimize these non- differentiable metrics, as rewards in a reinforce- ment learning paradigm. We also address the ex- posure bias issue by using a mixed-loss (Paulus et al., 2017; Wu et al., 2016), i.e., combining the cross-entropy and reward-based losses, which also helps maintain output fluency. Next, we introduce a novel entailment-corrected reward that checks for logically-directed partial matches. Current reinforcement-based text gener- ation works use traditional phrase-matching met- rics (e.g., CIDEr, BLEU) as their reward func- tion. However, these metrics use undirected n- gram matching of the machine-generated caption with the ground-truth caption, and hence fail to capture its directed logical correctness. Therefore, they still give high scores to even those generated captions that contain a single but critical wrong word (e.g., negation, unrelated action or object), because all the other words still match with the ground truth. We introduce CIDEnt, which pe- nalizes the phrase-matching metric (CIDEr) based reward, when the entailment score is low. This ensures that a generated caption gets a high re- 979 Ent CIDEr LSTM LSTM LSTM LSTM LSTM ... ... ... ... …

Content truncated to 500 words