A Unifying View On Task-oriented Dialogue Annotation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1286–1296 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1286 DIASER: A Unifying View On Task-oriented Dialogue Annotation Vojtˇech Hudeˇcek1, L´eon-Paul Schaub2,3, Daniel ˇStancl1, Patrick Paroubek2, Ondˇrej Duˇsek1 hudecek@ufal.mff.cuni.cz, schaub@limsi.fr, stancl@ufal.mff.cuni.cz, pap@limsi.fr, odusek@ufal.mff.cuni.cz 1Charles University, Faculty of Mathematics and Physics Malostransk´e n´amˇest´ı 25, 118 00 Prague, Czechia 2Universit´e Paris-Saclay, CNRS, Laboratoire Interdisciplinaire des Sciences du Num´erique 91400 Orsay, France 3Akio 43 rue de Dunkerque, 75010 Paris, France Abstract Every model is only as strong as the data that it is trained on. In this paper, we present a new dataset, obtained by merging four publicly available annotated corpora for task-oriented dialogues in several domains (MultiWOZ 2.2, CamRest676, DSTC2 and Schema-Guided Dialogue Dataset). This way, we assess the feasibility of providing a unified ontology and annotation schema covering several domains with a relatively limited effort. We analyze the characteristics of the resulting dataset along three main dimensions: language, information content and performance. We focus on aspects likely to be pertinent for improving dialogue success, e.g. dialogue consistency. Furthermore, to assess the usability of this new corpus, we thoroughly evaluate dialogue generation performance under various conditions with the help of two prominent recent end-to-end dialogue models: MarCo and GPT-2. These models were selected as popular open implementations representative of the two main dimensions of dialogue modelling. While we did not observe a significant gain for dialogue state tracking performance, we show that using more training data from different sources can improve language modelling capabilities and positively impact dialogue flow (consistency). In addition, we provide the community with one of the largest open dataset for machine learning experiments. Keywords: task oriented dialog, annotated corpora, resource merging 1. Introduction Recent research attention in task-oriented dialogue sys- tems focuses on end-to-end neural models. There have been many successful systems, including systems ca- pable of working with multiple domains (Lei et al., 2018; Mehri et al., 2019; Qin et al., 2020). These neural systems are notoriously data-hungry; especially with pretrained language models (LMs), their capacity vastly exceeds the amounts of data available with task- oriented annotation (Budzianowski and Vuli´c, 2019; Yang et al., 2021). There are several datasets with sim- ilar annotation style and similar domains (Serban et al., 2018), but they are often incompatible and training a model on a union of these datasets remains a challenge (Chen et al., 2019a). Despite recent efforts to provide support at the software level (Zhu et al., 2020b), data unification efforts have been limited so far. The aim of this paper is to address this issue and produce a large- scale open dataset with unified dialogue annotation. We call this new corpus DIASER (DIAlog System ex- tendER). To the best of our knowledge, no one has yet proposed a corpus as large as the one we have pro- duced by merging annotations from different corpora and domains under a common annotation scheme. We observe the effect of the increased size …

Content truncated to 500 words