The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9531 - 9539 December 7-11, 2022 ©2022 Association for Computational Linguistics The Aligned Multimodal Movie Treebank: An audio, video, dependency-parse treebank Adam Yaari MIT CSAIL–CBMM Jan DeWitt MIT CSAIL-CBMM Henry Hu MIT CSAIL-CBMM Bennett Stankovits MIT CSAIL-CBMM Sue Felshin MIT CSAIL-CBMM Yevgeni Berzak Technion Helena Aparicio Cornell University Boris Katz MIT CSAIL-CBMM Ignacio Cases* MIT CSAIL-CBMM Andrei Barbu* MIT CSAIL-CBMM Abstract Treebanks have traditionally included only text and were derived from written sources such as newspapers or the web. We intro- duce the Aligned Multimodal Movie Tree- bank (AMMT)†, an English language tree- bank derived from dialog in Hollywood movies which includes transcriptions of the audio- visual streams with word-level alignment, as well as part of speech tags and dependency parses in the Universal Dependencies (UD) for- malism. AMMT consists of 31,264 sentences and 218,090 words, that will amount to the 3rd largest UD English treebank and the only multi- modal treebank in UD. We find that parsers on this dataset often have difficulty with conversa- tional speech and that they often rely on punctu- ation which is often not available from speech recognizers. To help with the web-based an- notation effort, we also introduce the Efficient Audio Alignment Annotator (EAAA)‡, a com- panion tool that enables annotators to signifi- cantly speed-up their annotation processes. Keywords: multimodal, video, audio, tree- bank, Universal Dependency parsing Correspondence to {yaari,cases,abarbu}@mit.edu * Equal senior contribution. †https://github.com/abarbu/ammt ‡https://github.com/abarbu/audio-annotation 1 Introduction Treebanks are fundamental resources in Natural Language Processing (Nivre et al., 2016). Despite their central role, most existing treebanks are de- rived from single-modality texts such as newspa- pers, blogs, and other online communities. The vocabulary, syntax, and statistics of spoken and written language can be quite different from one another (Caines et al., 2017). To complement these datasets and aid the advent of multimodal conver- sational agents, we have created a new dataset, the Aligned Multimodal Movie Treebank, AMMT, the content of which is derived from language spoken in Hollywood movies. AMMT is released pub- licly under an open source license and will be con- tributed to the Universal Dependencies (UD) (Nivre et al., 2020) treebanks. Speech based treebanks have proven to be a resource of enormous importance to the NLP re- search community (Ahrenberg, 2007; Nivre et al., 2006). We find Treebank-3 of the Penn Treebank (Marcus et al., 1993), which includes the Penn Tree- bank Switchboard corpus (Godfrey et al., 1992), to be the closest existing dataset to AMMT. This cor- pus contains nearly one million transcribed words from Switchboard annotated with part of speech tags, dysfluencies, and parse trees, and it also in- Figure 1: An overview of AMMT, our novel multimodal dataset, consisting of transcriptions and parses for 21 movies aligned at the millisecond level. EAAA is a new transcription and alignment tool introduced below. 9531 cludes alignment between words and audio. How- ever, there are several key differences between this dataset and our own. AMMT provides …

Content truncated to 500 words