YouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 5133–5143, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 5133 YouMakeup: A Large-Scale Domain-Specific Multimodal Dataset for Fine-Grained Semantic Comprehension Weiying Wang Yongcheng Wang Shizhe Chen Qin Jin ∗ School of Information, Renmin Unversity of China {wy.wang, 2015202012, cszhe1, qjin}@ruc.edu.cn Abstract Multimodal semantic comprehension has at- tracted increasing research interests in recent years, such as visual question answering and caption generation. However, due to the data limitation, fine-grained semantic comprehen- sion which requires to capture semantic de- tails of multimodal contents has not been well investigated. In this work, we intro- duce “YouMakeup”, a large-scale multimodal instructional video dataset to support fine- grained semantic comprehension research in specific domain. YouMakeup contains 2,800 videos from YouTube, spanning more than 420 hours in total. Each video is annotated with a sequence of natural language descriptions for instructional steps, grounded in temporal video range and spatial facial areas. The anno- tated steps in a video involve subtle difference in actions, products and regions, which require fine-grained understanding and reasoning both temporally and spatially. In order to evaluate models’ ability for fined-grained comprehen- sion, we further propose two groups of tasks including generation tasks and visual question answering tasks from different aspects. We also establish a baseline of step caption gen- eration for future comparison. The dataset will be publicly available at https:// github.com/AIM3-RUC/YouMakeup to support research investigation in fine-grained semantic comprehension. 1 Introduction Videos which naturally contain rich multimodal semantic information have been one of the main sources for knowledge acquisition. In recent years, video semantic comprehension has attracted much research attention, with a number of datasets and tasks being proposed such as activity recog- nition (Xu et al., 2017), dense video captioning ∗ Corresponding author. (Krishna et al., 2017a), visual question answering (Lei et al., 2018, 2019) etc. However, most works are limited to only capturing coarse semantic in- formation such as action recognition in broad cat- egories. Fine-grained comprehension instead has not been fully explored, especially for discriminat- ing actions with subtle difference or understanding temporal relations of actions in a certain activity. Instructional videos, which contain series of steps to accomplish certain tasks, are suitable sources to investigate fine-grained semantic com- prehension and reasoning. As shown in Table 1, different instructional video datasets have been released. However, current datasets suffer from small data scales or coarse annotations to support fine-grained analysis. For example, datasets col- lected in (Alayrac et al., 2016; Rohrbach et al., 2012; Stein and McKenna, 2013; Kuehne et al., 2014; Das et al., 2013) only contain hundreds or fewer videos and actions. Although COIN dataset (Tang et al., 2019) is in large scale, it aims to cover wide range of action categories in- stead of distinguishing actions with subtle differ- ence, and also lacks fine-grained step annotations. The YouCook2 dataset (Zhou et al., 2018) con- tains fairly large number of videos …

Content truncated to 500 words