Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation

Cross-Lingual Dialogue Dataset Creation via Outline-Based Generation Olga Majewska⋄ Evgeniia Razumovskaia⋄ Edoardo M. Ponti†⋄ Ivan Vuli´c⋄ Anna Korhonen⋄ ⋄Language Technology Lab, University of Cambridge, United Kingdom †Institute for Language, Cognition and Computation, University of Edinburgh, United Kingdom {om304,er563,ep490,iv250,alk23}@cam.ac.uk Abstract Multilingual task-oriented dialogue (ToD) fa- cilitates access to services and information for many (communities of) speakers. Neverthe- less, its potential is not fully realized, as current multilingual ToD datasets—both for modular and end-to-end modeling—suffer from severe limitations. 1) When created from scratch, they are usually small in scale and fail to cover many possible dialogue flows. 2) Translation-based ToD datasets might lack naturalness and cul- tural specificity in the target language. In this work, to tackle these limitations we propose a novel outline-based annotation pro- cess for multilingual ToD datasets, where domain-specific abstract schemata of dialogue are mapped into natural language outlines. These in turn guide the target language an- notators in writing dialogues by providing instructions about each turn’s intents and slots. Through this process we annotate a new large-scale dataset for evaluation of multi- lingual and cross-lingual ToD systems. Our Cross-lingual Outline-based Dialogue dataset (COD) enables natural language understand- ing, dialogue state tracking, and end-to-end dialogue evaluation in 4 diverse languages: Arabic, Indonesian, Russian, and Kiswahili. Qualitative and quantitative analyses of COD versus an equivalent translation-based dataset demonstrate improvements in data quality, un- locked by the outline-based approach. Finally, we benchmark a series of state-of-the-art sys- tems for cross-lingual ToD, setting reference scores for future work and demonstrating that COD prevents over-inflated performance, typ- ically met with prior translation-based ToD datasets. 1 Introduction and Motivation One of the staples of machine intelligence is the ability to communicate with humans and complete a task as instructed during such an interaction. This is commonly referred to as task-oriented dialogue (ToD; Gupta et al., 2005; Bohus and Rudnicky 2009; Young et al., 2013; Muise et al., 2019). Despite having far-reaching applications, such as banking (Altinok, 2018), travel (Zang et al., 2020), and healthcare (Denecke et al., 2019), this technology is currently accessible to very few communities of speakers (Razumovskaia et al., 2022a). The progress in multilingual ToD is critically hampered by the paucity of training data for many of the world’s languages. While cross-lingual transfer learning (Zhang et al., 2019; Xu et al., 2020; Siddhant et al., 2020; Krishnan et al., 2021) offers a partial remedy, its success is tenuous beyond typologically similar languages and gen- erally hard to assess due to the lack of evaluation benchmarks (Razumovskaia et al., 2022a). What is more, transfer learning often cannot leverage multi-source transfer and few-shot learning due to lack of language diversity in the ToD datasets (Zhu et al., 2020; Quan et al., 2020; Farajian et al., 2020). Therefore, the main driver of development in multilingual ToD is the creation of multilingual resources. However, even when available, they suffer from several pitfalls. Most are obtained by manual or semi-automatic translation of an En- glish source (Castellucci et al., 2019; Bellomaria et al., 2019; Susanto …

Content truncated to 500 words