Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1961–1966, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Improving LSTM-based Video Description with Linguistic Knowledge Mined from Text Subhashini Venugopalan UT Austin vsub@cs.utexas.edu Lisa Anne Hendricks UC Berkeley lisa anne@berkeley.edu Raymond Mooney UT Austin mooney@cs.utexas.edu Kate Saenko Boston University saenko@bu.edu Abstract This paper investigates how linguistic knowl- edge mined from large text corpora can aid the generation of natural language descriptions of videos. Specifically, we integrate both a neu- ral language model and distributional seman- tics trained on large text corpora into a recent LSTM-based architecture for video descrip- tion. We evaluate our approach on a collection of Youtube videos as well as two large movie description datasets showing significant im- provements in grammaticality while modestly improving descriptive quality. 1 Introduction The ability to automatically describe videos in nat- ural language (NL) enables many important appli- cations including content-based video retrieval and video description for the visually impaired. The most effective recent methods (Venugopalan et al., 2015a; Yao et al., 2015) use recurrent neural net- works (RNN) and treat the problem as machine translation (MT) from video to natural language. Deep learning methods such as RNNs need large training corpora; however, there is a lack of high- quality paired video-sentence data. In contrast, raw text corpora are widely available and exhibit rich linguistic structure that can aid video description. Most work in statistical MT utilizes both a language model trained on a large corpus of monolingual tar- get language data as well as a translation model trained on more limited parallel bilingual data. This paper explores methods to incorporate knowledge from language corpora to capture general linguistic regularities to aid video description. This paper integrates linguistic information into a video-captioning model based on Long Short Term Memory (LSTM) (Hochreiter and Schmidhu- ber, 1997) RNNs which have shown state-of-the-art performance on the task. Further, LSTMs are also effective as language models (LMs) (Sundermeyer et al., 2010). Our first approach (early fusion) is to pre-train the network on plain text before train- ing on parallel video-text corpora. Our next two ap- proaches, inspired by recent MT work (Gulcehre et al., 2015), integrate an LSTM LM with the existing video-to-text model. Furthermore, we also explore replacing the standard one-hot word encoding with distributional vectors trained on external corpora. We present detailed comparisons between the ap- proaches, evaluating them on a standard Youtube corpus and two recent large movie description datasets. The results demonstrate significant im- provements in grammaticality of the descriptions (as determined by crowdsourced human evaluations) and more modest improvements in descriptive qual- ity (as determined by both crowdsourced human judgements and standard automated comparison to human-generated descriptions). Our main contribu- tions are 1) multiple ways to incorporate knowledge from external text into an existing captioning model, 2) extensive experiments comparing the methods on three large video-caption datasets, and 3) human judgements to show that external linguistic knowl- edge has a significant impact on grammar. 2 …

Content truncated to 500 words