BehanceCC: A ChitChat Detection Dataset For Livestreaming Video Transcripts

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 7284–7290 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 7284 BehanceCC: A ChitChat Detection Dataset For Livestreaming Video Transcripts Viet Dac Lai1, Amir Pouran Ben Veyseh1, Franck Dernoncourt2, Thien Huu Nguyen1 1Dept. of Computer and Information Science, University of Oregon, Eugene, OR, USA 2Adobe Research, Seattle, WA, USA {vietl,apouranb,thien}@cs.uoregon.edu, franck.dernoncourt@adobe.com Abstract Livestreaming videos have become an effective broadcasting method for both video sharing and educational purposes. However, livestreaming videos contain a considerable amount of off-topic content (i.e., up to 50%) which introduces significant noises and data load to downstream applications. This paper presents BehanceCC, a new human-annotated benchmark dataset for off-topic detection (also called chitchat detection) in livestreaming video transcripts. In addition to describing the challenges of the dataset, our extensive experiments of various baselines reveal the complexity of chitchat detection for livestreaming videos and suggest potential future research directions for this task. The dataset will be made publicly available to foster research in this area. The dataset is freely accessible at https://github.com/nlp-uoregon/behancecc. Keywords: Chitchat Detection, Livestreaming Video Transcripts 1. Introduction Livestreaming is becoming an essential communication medium in human life to connect people around the world. Content creators and audiences have been us- ing livestreaming platforms for various purposes, in- cluding video sharing (Youtube Live, and Facebook Live), gaming (Twitch), entertainment (TikTok), and online learning (Behance). Compared to the origi- nal design for most of these platforms to share pre- recorded videos, livestreaming introduces a new impor- tant feature that allow content creators to connect with their audiences in real-time, thus making the platforms more realistic and useful. In fact, its engagement effi- ciency has promoted livestreaming as one of the most important mechanisms to attract and retain content cre- ators and audiences in current social platforms. However, the shift from pre-recorded videos to livestreaming videos, which leads to an entire change of production processes, creates a major problem in the quality control of the produced videos. In partic- ular, in pre-recording production, video content is usu- ally well prepared and carefully post-edited. Therefore, pre-recorded videos are usually shorter (i.e., from few minutes and up to a few hours), and involve concise and focused content, fewer verbal pauses, and no audience interference. In contrast, in livestreaming production, video content is streaming live to some chosen plat- forms. It allows the speakers to interact with audiences in real-time through chat box or even live discussion, thus offering an effective tool to gain more exposure and engagement through the video content. Thus, live broadcasts introduce many unfavorable qual- ity concerns. First, livestreaming videos tend to be much longer than pre-recorded videos as post-editing and cutting are not performed in livestreaming videos. Second, speakers in livestreaming videos tend to use a lot of verbal pauses and word/phrase repetitions as in casual discussion (e.g., due to poor preparation, think- Figure 1: Chitchat texts in a livestreaming video tran- script (highlighted in orange). ing, and hesitation). …

Content truncated to 500 words