Semi-Supervised Learning for Video Captioning
32381Authors
nan
Abstract
AbstractDeep neural networks have made great success on video captioning in supervised learning setting. However, annotating videos with descriptions is very expensive and time-consuming. If the video captioning algorithm can benefit from a large number of unlabeled videos, the cost of annotation can be reduced. In the proposed study, we make the first attempt to train the video captioning model on labeled data and unlabeled data jointly, in a semi-supervised learning manner. For labeled data, we train them with the traditional cross-entropy loss. For unlabeled data, we leverage a self-critical policy gradient method with the difference between the scores obtained by Monte-Carlo sampling and greedy decoding as the reward function, while the scores are the negative K-L divergence between output distributions of original video data and augmented video data. The final loss is the weighted sum of losses obtained by labeled data and unlabeled data. Experiments conducted on VATEX, MSR-VTT and MSVD dataset demonstrate that the introduction of unlabeled data can improve the performance of the video captioning model. The proposed semi-supervised learning algorithm also outperforms several state-of-the-art semi-supervised learning approaches.
Keywords
Full Content
Findings of the Association for Computational Linguistics: EMNLP 2020, pages 94–108 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 94 GRUEN for Evaluating Linguistic Quality of Generated Text Wanzheng Zhu and Suma Bhat University of Illinois at Urbana-Champaign, USA wz6@illinois.edu, spbhat2@illinois.edu Abstract Automatic evaluation metrics are indispens- able for evaluating generated text. To date, these metrics have focused almost exclusively on the content selection aspect of the system output, ignoring the linguistic quality aspect altogether. We bridge this gap by proposing GRUEN for evaluating Grammaticality, non- Redundancy, focUs, structure and coherENce of generated text.1 GRUEN utilizes a BERT- based model and a class of syntactic, seman- tic, and contextual features to examine the sys- tem output. Unlike most existing evaluation metrics which require human references as an input, GRUEN is reference-less and requires only the system output. Besides, it has the ad- vantage of being unsupervised, deterministic, and adaptable to various tasks. Experiments on seven datasets over four language genera- tion tasks show that the proposed metric corre- lates highly with human judgments.2 1 Introduction Automatic evaluation metrics for Natural Language Generation (NLG) tasks reduce the need for hu- man evaluations, which can be expensive and time- consuming to collect. Fully automatic metrics al- low faster measures of progress when training and testing models, and therefore, accelerate the devel- opment of NLG systems (Chaganty et al., 2018; Zhang et al., 2020; Clark et al., 2019). To date, most automatic metrics have focused on measuring the content selection between the human references and the model output, leaving linguistic quality to be only indirectly captured (e.g., n-gram and longest common subsequence in ROUGE-N and ROUGE-L respectively (Lin and Hovy, 2003; 1Following BLEU and ROUGE – blue and red in French, we name our evaluation metric GRUEN – that means green in German. 2Our metric is available at https://github.com/ WanzhengZhu/GRUEN. Q1: Grammaticality The summary should have no date- lines, system-internal formatting, capitalization errors or obviously ungrammatical sentences (e.g., fragments, miss- ing components) that make the text difficult to read. Q2: Non-redundancy There should be no unnecessary repetition in the summary. Q3: Focus The summary should have a focus; sentences should only contain information that is related to the rest of the summary. Q4: Structure and Coherence The summary should be well-structured and well-organized. The summary should not just be a heap of related information, but should build from sentence to sentence to a coherent body of information about a topic. Table 1: Dimensions of linguistic quality as proposed in Dang (2006). Lin, 2004), and alignment in METEOR (Baner- jee and Lavie, 2005)). Even though the need for an explicit measure of linguistic quality has long been pointed out in Dang (2006); Conroy and Dang (2008), this aspect has remained under-explored barring a few studies that focused on measuring the linguistic quality of a generated piece of text (Pitler et al., 2010; Kate et al., 2010; Xenouleas et al., 2019). In this paper, we bridge this gap by proposing a novel metric …
Content truncated to 500 words
Labeled Text
AbstractDeep neural networks have made great success on video captioning in supervised learning setting. However, annotating videos with descriptions is very expensive and time-consuming. If the video captioning algorithm can benefit from a large number of unlabeled videos, the cost of annotation can be reduced. In the proposed study, we make the first attempt to train the video captioning model on labeled data and unlabeled data jointly, in a semi-supervised learning manner. For labeled data, we train them with the traditional cross-entropy loss. For unlabeled data, we leverage a self-critical policy gradient method with the difference between the scores obtained by Monte-Carlo sampling and greedy decoding as the reward function, while the scores are the negative K-L divergence between output distributions of original video data and augmented video data. The final loss is the weighted sum of losses obtained by labeled data and unlabeled data. Experiments conducted on VATEX, MSR-VTT and MSVD dataset demonstrate that the introduction of unlabeled data can improve the performance of the video captioning model. The proposed semi-supervised learning algorithm also outperforms several state-of-the-art semi-supervised learning approaches.
Publication Details
- Paper ID:
- 32381
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan