TutorialVQA: Question Answering Dataset for Tutorial Videos

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 5450–5455 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 5450 TutorialVQA: Question Answering Dataset for Tutorial Videos Anthony Colas∗, Seokhwan Kim†, Franck Dernoncourt†, Siddhesh Gupte∗, Daisy Zhe Wang∗, Doo Soon Kim† *University of Florida, † Adobe Research * 432 Newell Dr, Gainesville, FL 32611, † 345 Park Avenue San Jose, CA 95110-2704 *{acolas1, daisyw, siddhesh.gupte}@ufl.edu †{seokim, dernonco,dkim}@adobe.com Abstract Despite the number of currently available datasets on video-question answering, there still remains a need for a dataset involving multi-step and non-factoid answers. Moreover, relying on video transcripts remains an under-explored topic. To adequately address this, we propose a new question answering task on instructional videos, because of their verbose and narrative nature. While previous studies on video question answering have focused on generating a short text as an answer, given a question and video clip, our task aims to identify a span of a video segment as an answer which contains instructional details with various granularities. This work focuses on screencast tutorial videos pertaining to an image editing program. We introduce a dataset, TutorialVQA, consisting of about 6,000 manually collected triples of (video, question, answer span). We also provide experimental results with several baseline algorithms using the video transcripts. The results indicate that the task is challenging and call for the investigation of new algorithms. Keywords: question answering, resources, evaluation, corpora 1. Introduction Video is the fastest growing medium to create and deliver information today. Consequentially, videos have been in- creasingly used as main data sources in many question an- swering problems (Yang et al., 2003; Tapaswi et al., 2016; Jang et al., 2017; Maharaj et al., 2017; Kim et al., 2017; Jang et al., 2017; Zeng et al., 2017). These previous studies have mostly focused on factoid questions, each of which can be answered in a few words or phrases generated by understanding multimodal contents in a short video clip. However, this problem definition of video question answer- ing causes some practical limitations for the following rea- sons. First, factoid questions are just a small part of what people actually want to ask on video contents. Especially if a short video is given to users, most fragmentary facts within the scope of previous tasks can be easily perceived by themselves even before asking questions. Thus, video question answering is expected to provide answers to more complicated non-factoid questions beyond the simple facts. For example, users may have “how to” type questions where the answer involves many fragmented steps to com- plete the task, as shown in Fig. 2. Accordingly, the answer format needs to also be im- proved towards more flexible ways besides multiple choice (Tapaswi et al., 2016; Jang et al., 2017) or fill-in- the-blank questions (Maharaj et al., 2017; Kim et al., 2017). Although open-ended video question answering (Yang et al., 2003; Jang et al., 2017; Zeng et al., 2017) has been ex- plored, it still aims to generate just …

Content truncated to 500 words