A hierarchical approach to vision-based language generation: from simple sentences to complex natural language

Proceedings of the 28th International Conference on Computational Linguistics, pages 2436–2447 Barcelona, Spain (Online), December 8-13, 2020 2436 A hierarchical approach to vision-based language generation: from simple sentences to complex natural language Simion-Vlad Bogolin* Ioana Croitoru* Marius Leordeanu Institute of Mathematics of the Romanian Academy University ”Politehnica” of Bucharest vladbogolin@gmail.com ioana.croi@gmail.com marius.leordeanu@imar.ro Abstract Automatically describing videos in natural language is an ambitious problem, which could bridge our understanding of vision and language. We propose a hierarchical approach, by first gener- ating video descriptions as sequences of simple sentences, followed at the next level by a more complex and fluent description in natural language. While the simple sentences describe simple actions in the form of (subject, verb, object), the second-level paragraph descriptions, indirectly using information from the first-level description, presents the visual content in a more com- pact, coherent and semantically rich manner. To this end, we introduce the first video dataset in the literature that is annotated with captions at two levels of linguistic complexity. We perform extensive tests that demonstrate that our hierarchical linguistic representation, from simple to complex language, allows us to train a two-stage network that is able to generate significantly more complex paragraphs than current one-stage approaches. 1 Introduction Automatic video to text translation is a very difficult, still unsolved problem, at the intersection of natural language processing, machine learning and computer vision which has been widely researched in the last years (Aafaq et al., 2019). Its applications are varied and far reaching, with impact in virtually all aspects of our lives. One of the main challenges is that for a given visual input, there are infinitely many ways to describe it in natural language. Every human utterance is unique (Chomsky and Lightfoot, 2002; Pinker, 2003), meaning that, while the classic machine learning paradigm assumes, for a given input, a certain ”label” from a finite set, language is much more complex and versatile. Recurrent neural networks (RNN) establish the current basis for research in video to language transla- tion. Using Long-Short Term Memory (LSTM) (Hochreiter and Schmidhuber, 1997) to produce the final text description has become the standard approach. The most common architecture is the sequence-to- sequence model (Venugopalan et al., 2015; Donahue et al., 2015), which usually produces good results. However, it is still unclear what the neural network learns, w.r.t language representation, semantics and diversity. Recent works (Shetty et al., 2017) have clearly pointed out that automated systems have a significantly less diverse vocabulary than humans. It has also been shown that as the sentence becomes longer, the relation to the actual video content drops (Shetty et al., 2017). Our goal is to study whether a dual, intermediate linguistic description at the level of short sentences, describing simple actions in the form of (subject, verb, object) could help in the generation of more com- plex and fluent language. Common sense and the challenges encountered in current research, strongly suggest that the transition from vision to language could benefit from a more gradual approach: we …

Content truncated to 500 words