EZCAT: an Easy Conversation Annotation Tool

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1788–1797 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1788 EZCAT: an Easy Conversation Annotation Tool Ga¨el Guibona,b, Luce Lefeuvreb, Matthieu Labeaua, Chlo´e Clavela a LTCI, T´el´ecom-Paris, Institut Polytechnique de Paris, b Direction Technologies, Innovation & Projets Groupe, SNCF 19 place marguerite perey, 91120 Palaiseau; 1/3 Av. Franc¸ois Mitterrand, 93210 Saint-Denis {gael.guibon, matthieu.labeau, chloe.clavel}@telecom-paris.fr {luce.lefeuvre, ext.gael.guibon}@sncf.fr Abstract Users generate content constantly, leading to new data requiring annotation. Among this data, textual conversations are created every day and come with some specificities: they are mostly private through instant messaging applications, requiring the conversational context to be labeled. These specificities led to several annotation tools dedicated to conversation, and mostly dedicated to dialogue tasks, requiring complex annotation schemata, not always customizable and not taking into account conversation-level labels. In this paper, we present EZCAT, an easy-to-use interface to annotate conversations in a two-level configurable schema, leveraging message-level labels and conversation-level labels at once. Our interface is characterized by the voluntary absence of a server and accounts management, enhancing its availability to anyone, and the control over data, which is crucial to confidential conversations. We also present our first usage of EZCAT along with our annotation schema we used to annotate confidential customer service conversations. EZCAT is freely available at https://gguibon.github.io/ezcat. Keywords: conversations, annotation tool, text messages 1. Introduction In the recent years, text have been one of the main user generated content modality, whether it is a comment, a blog, or chats. Among these, textual conversations led to many studies on dialogue (Zhu et al., 2019; Ma et al., 2020; Colombo et al., 2020; Deriu et al., 2021; Razu- movskaia et al., 2021), which could be multi-modal (Nie et al., 2019; Chapuis et al., 2020) or specialized on a modality such as text. These studies have been possi- ble thanks to the rise of available public corpora dedi- cated to dialogue (Jurafsky et al., 1997; Shriberg et al., 1998; Stolcke et al., 2000; Novikova et al., 2017; Li et al., 2017). However, this improvement only concerns a part of the User Generated Content (UGC), and private dyadic conversations are yet to be found. These con- versations usually come from messaging applications such as WhatsApp, Telegram, or WeChat, to name but a few. Due to privacy concerns, sharing them is not easy; but many dialogue systems would benefit from performance reports obtained on real data. This re- quires annotated data sets, and therefore tools for con- versation annotation. We propose a quick, easy-to-use, and freely accessible annotation tool dedicated to spon- taneous conversations. This tool aims at reducing annotation costs and to en- abling short annotation tasks with simple custom anno- tation schemata for anyone. It is designed to quickly annotate private conversation data in order, for in- stance, to have another point of view of the perfor- mance of a model on a real-data test set. We follow the global tendency in Natural Language …

Content truncated to 500 words