The slurk Interaction Server Framework: Better Data for Better Dialog Models

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 4069–4078 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 4069 The slurk Interaction Server Framework: Better Data for Better Dialog Models Jana G¨otze, Maike Paetzel-Pr¨usmann, Wencke Liermann, Tim Diekmann, David Schlangen Foundations of Computational Linguistics, Department Linguistics University of Potsdam, Germany {firstname.lastname}@uni-potsdam.de, diekmann.tim@gmail.com Abstract This paper presents the slurk software, a lightweight interaction server for setting up dialog data collections and running experiments. slurk enables a multitude of settings including text-based, speech and video interaction between two or more humans or humans and bots, and a multimodal display area for presenting shared or private interactive context. The software is implemented in Python with an HTML and JAVASCRIPT frontend that can easily be adapted to individual needs. It also provides a setup for pairing participants on common crowdworking platforms such as Amazon Mechanical Turk and some example bot scripts for common interaction scenarios. Keywords: dialog, data collection tool, interaction, chat, spoken dialog, multimodal dialog, crowdsourcing 1. Introduction Much of NLP’s breakthroughs in recent years is based on data-driven learning methods. Data-hungry ma- chine learning algorithms to model Visual Question Answering (Das et al., 2017) or Vision and Language Navigation Tasks (Krantz et al., 2020) are fed with crowdsourced data, which is fast and affordable to ob- tain, as crowdworkers do not have to be brought to the lab for common tasks like labeling, captioning im- ages or producing navigation instructions. Data for the subfield of dialog modelling requires at least two crowdworkers to be involved, something that is a non- standard use case for the most popular crowdsourcing platforms as it requires coordinating two or more work- ers to find a common timeslot to work on a task. The slurk tool adds to a body of frameworks and tools (Healey et al., 2003; Manuvinakurike and De- Vault, 2015; Miller et al., 2017; Schlangen et al., 2018) that facilitate data collection for training and testing di- alog models where it is often necessary for researchers to set up their own data collection that satisfies their specific needs, e.g., to cover some specific domain or dialog phenomenon. slurk allows to create exper- iments for human-human or human-machine interac- tion, with no limitation on the number of participants for a given setup. Possible interaction channels include text as well as audio and video. Dialog games – con- sisting of an interaction setting such as a text or audio channel, a certain context to refer to, and a task to solve – can be created to include multimodal context such as images or interactive tools in which participants can manipulate the context together. In this paper, we describe the slurk software, a mod- ular tool for collecting multimodal dialog data that is integratable with crowdworking platforms to pair up participants on demand. We explain how slurk can be set up to constrain the interaction channel in a num- ber of ways as well as to manipulate the …

Content truncated to 500 words