Seshat: a Tool for Managing and Verifying Annotation Campaigns of Audio Data

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 6976–6982 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 6976 Seshat: A tool for managing and verifying annotation campaigns of audio data Hadrien Titeux⋆1 , Rachid Riad⋆1,2, Xuan-Nga Cao1, Nicolas Hamilakis1, Kris Madden1, Alejandrina Cristia1, Anne-Catherine Bachoud-L´evi2, Emmanuel Dupoux1 1 LSCP/ENS/CNRS/EHESS/INRIA/PSL Research University, Paris, France 2 NPI/ENS/INSERM/UPEC/PSL Research University, Cr´eteil, France {hadrien.titeux, rachid.riad}@ens.fr, {ngafrance, nick.hamilakis562, thekrismadden, alecristia, bachoud, emmanuel.dupoux}@gmail.com Abstract We introduce Seshat, a new, simple and open-source software to efficiently manage annotations of speech corpora. The Seshat software allows users to easily customise and manage annotations of large audio corpora while ensuring compliance with the formatting and naming conventions of the annotated output files. In addition, it includes procedures for checking the content of annotations following specific rules that can be implemented in personalised parsers. Finally, we propose a double-annotation mode, for which Seshat computes automatically an associated inter-annotator agreement with the γ measure taking into account the categorisation and segmentation discrepancies. Keywords: speech transcription, speech corpora, annotations management 1. Introduction Large corpora of speech, obtained in the laboratory and in naturalistic conditions, become easier to collect. This new trend broadens the scope of scientific questions on speech and language that can be answered. However, this poses an important challenge for the construction of reliable and usable annotations. Managing annotators and ensur- ing the quality of their annotations are highly demanding tasks for research endeavours and industrial projects (Zue et al., 1990). When organised manually, the manager of annotation campaigns usually faces three major problems: the mishandling of files (e.g., character-encoding problems, incorrect naming of files), the non-conformity of the anno- tations (Moreno et al., 2000), and the inconsistency of the annotations (Gut and Bayerl, 2004). In this paper, we introduce Seshat, a system for the au- tomated management of annotation campaigns for au- dio/speech data which addresses these challenges. It is built on two components that communicate via a Restful API: a back-end (server) written in Flask and a front-end (client) in Angular Typescript. Seshat is easy to install for non- developers and easy to use for researchers and annotators while having some extension capabilities for developers. In Section 2, we describe the related work on annotations tools, which do not provide solutions to all the aforemen- tioned challenges during corpus creation. In Section 3, we make an overview of the different functionalities of the soft- ware. Then, we explain, in Section 4, the architecture of the software, and also the several UX/UI design and engineer- ing choices that have been made to facilitate the usage of the platform. We describe how to use of Seshat in Section 5 and Section 6 presents two specific use-cases. Finally, we conclude and describe future plans for Seshat in Section 7. ⋆ Equal contribution. This work was conducted while E. Dupoux was a part-time Research Scientist at Facebook AI Re- search. Code for Seshat is available on Github at https:// github.com/bootphon/seshat 2. Related …

Content truncated to 500 words