Video Caption Dataset for Describing Human Actions in Japanese

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4664–4670 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4664 Video Caption Dataset for Describing Human Actions in Japanese Yutaro Shigeto, Yuya Yoshikawa, Jiaqing Lin, Akikazu Takeuchi STAIR Lab, Chiba Institute of Technology Tsudanuma 2-17-1, Narashino, Chiba, Japan {shigeto, yoshikawa, lin, takeuchi}@stair.center Abstract In recent years, automatic video caption generation has attracted considerable attention. This paper focuses on the generation of Japanese captions for describing human actions. While most currently available video caption datasets have been constructed for English, there is no equivalent Japanese dataset. To address this, we constructed a large-scale Japanese video caption dataset consisting of 79,822 videos and 399,233 captions. Each caption in our dataset describes a video in the form of “who does what and where.” To describe human actions, it is important to identify the details of a person, place, and action. Indeed, when we describe human actions, we usually mention the scene, person, and action. In our experiments, we evaluated two caption generation methods to obtain benchmark results. Further, we investigated whether those generation methods could specify “who does what and where.” Keywords: video captioning, caption generation, Japanese caption dataset, human action understanding 1. Introduction Automatic video caption generation is a task that outputs the description, or caption, of an input video (Venugopalan et al., 2015b; Venugopalan et al., 2015a; Yao et al., 2015). Video caption generation has many practical applications such as video searching using natural language queries, natural language video summarization, and use as a communi- cation robot. It can also be useful for visually impaired people. This paper tackles one of such application, namely auto- matic human action description generation. To describe human actions, it is necessary to recognize and under- stand “who does what and where.” When we explain hu- man actions, it is important to include details of the per- son, place, and action. While various researchers have al- ready introduced video caption datasets for describing hu- man actions (Sigurdsson et al., 2016; Krishna et al., 2017; Awad et al., 2018), none of these datasets evaluate those as- pects individually. Another problem is that there are difficulties in caption- ing resource-poor languages. Most previous research has focused on English caption generation due to the scarcity of resources targeting other languages. Each language is unique in terms of properties such as grammar and multi- word expressions, making it difficult to determine how to generate captions in other languages. Conversely, the prac- tical applications of caption generation are common to all languages. Hence there is massive demand for caption gen- eration in languages other than English. These issues motivated us to develop a video caption dataset for describing human actions in Japanese (Sect. 2). This dataset is based on 79,822 videos collected from STAIR Actions, a dataset for human action recogni- tion (Yoshikawa et al., 2018). Each video has five descrip- tions on average, resulting in 399,233 captions in total. Each caption specifies …

Content truncated to 500 words