The Multimodal Annotation Software Tool (MAST)

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 6822–6828 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 6822 The Multimodal Annotation Software Tool (MAST) Bruno Cardoso and Neil Cohn Tilburg University, Tilburg School of Humanities and Digital Sciences Department of Communication and Cognition P.O. Box 90153, 5000 LE Tilburg The Netherlands bruno.cardoso@tilburguniversity.edu, neilcohn@visuallanguagelab.com Abstract Multimodal combinations of writing and pictures have become ubiquitous in contemporary society, and scholars have increasingly been turning to analyzing these media. Here we present a resource for annotating these complex documents: the Multimodal Annotation Software Tool (MAST). MAST is an application that allows users to analyze visual and multimodal documents by selecting and annotating visual regions, and to establish relations between annotations that create dependencies and/or constituent structures. By means of schema publications, MAST allows annotation theories to be citable, while evolving and being shared. Documents can be annotated using multiple schemas simultaneously, offering more comprehensive perspectives. As a distributed, client-server system MAST allows for collaborative annotations across teams of users, and features team management and resource access functionalities, facilitating the potential for implementing open science practices. Altogether, we aim for MAST to provide a powerful and innovative annotation tool with application across numerous fields engaging with multimodal media. Keywords: multimodal annotation, annotation tool, multimodality, visual language, corpus linguistics 1. Introduction Contemporary society has seen a proliferation of media that combine writing with graphics, whether it is in comics, diagrams, memes, text/emoji, or many other contexts. Complementing this, scholars of language have become increasingly sensitive to analyzing these multimodal documents and creating multimodal corpora. Various annotation software tools have been created to carry out research on written/graphic multimodal documents, with many tied to the constraints of specific studies. However, no “standard” annotation software has yet emerged for these purposes, and none balance the principles of collaboration and flexibility we view as essential for contemporary research. Existing visual and multimodal annotation systems provide a range of tools, but are often limited in their scope and applicability. Often, unique annotation software has been designed for specific studies, such as for the study of comics (Dunst, Hartel, & Laubrock, 2017), instruction manuals (van der Sluis, Kloppenburg, & Redeker, 2016), or diagrams (Alikhani & Stone, 2018; Hiippala et al., 2020), among others. These tools are often designed around specific annotation theories, restricting users to work with predefined, built-in sets of annotations, and thus have limited applicability beyond their specific domains. To our knowledge, only a limited number of general, multipurpose tools have been designed for multimodal or visual annotation. For example, the UAM ImageTool (O’Donnell, 2008), allows for rectangular regions of interest to be drawn and annotated with user-defined annotation schemas. However, this application is limited to single users, requiring the sharing of saved files to facilitate collaboration. Additionally, UAM ImageTool restricts users to a single schema at a time to annotate documents, and does not let users specify relations between annotations for dependencies. Also, it is worth considering …

Content truncated to 500 words