Code-switched inspired losses for spoken dialog representations

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8320–8337 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 8320 Code-switched inspired losses for generic spoken dialog representations Emile Chapuis1∗, Pierre Colombo1,2∗, Matthieu Labeau1, Chloe Clavel1 1LTCI, Telecom Paris, Institut Polytechnique de Paris, 2IBM GBS France 1firstname.lastname@telecom-paris.fr, 2pierre.colombo@ibm.com Abstract Spoken dialog systems need to be able to han- dle both multiple languages and multilingual- ity inside a conversation (e.g in case of code- switching). In this work, we introduce new pretraining losses tailored to learn multilingual spoken dialog representations. The goal of these losses is to expose the model to code- switched language. To scale up training, we automatically build a pretraining corpus com- posed of multilingual conversations in five dif- ferent languages (French, Italian, English, Ger- man and Spanish) from OpenSubtitles, a huge multilingual corpus composed of 24.3G tokens. We test the generic representations on MIAM, a new benchmark composed of five di- alog act corpora on the same aforementioned languages as well as on two novel multilingual downstream tasks (i.e multilingual mask utter- ance retrieval and multilingual inconsistency identification). Our experiments show that our new code switched-inspired losses achieve a better performance in both monolingual and multilingual settings. 1 Introduction A crucial step in conversational AI is the identifica- tion of underlying information of the user’s utter- ance (e.g communicative intent or dialog acts, and emotions). This requires modeling utterance-level information (Mitkov, 2014; Williams et al., 2014), to capture immediate nuances of the user utterance; and discourse-level features (Thornbury and Slade, 2006), to capture patterns over long ranges of the conversation. An added difficulty to this modeling problem is that most people in the world are bilin- gual (Grosjean and Li, 2013): therefore, progress on these systems is limited by their inability to pro- cess more than one language (English being the most frequent). For example, many people use English as a “workplace" language but seamlessly ∗stands for equal contribution switch to their native language when the condi- tions are favorable (Heredia and Altarriba, 2001). Thus, there is a growing need for understanding dialogs in a multilingual fashion (Ipsic et al., 1999; Joshi et al., 2020; Ruder et al., 2019). Additionally, when speakers share more than one language, they inevitably will engage in code-switching (Sankoff and Poplack, 1981; Gumperz, 1982; Milroy et al., 1995; Auer, 2013; Parekh et al., 2020): switching between two different languages. Thus, spoken dialog systems need to be cross lingual (i.e able to handle different languages) but also need to model multilinguality inside a conversation (Ahn et al., 2020). In this paper, we focus on building generic repre- sentations for dialog systems that satisfy the afore- mentioned requirements. Generic representations have led to strong improvements on numerous nat- ural language understanding tasks, and can be fine- tuned when only small labelled datasets are avail- able for the desired downstream task (Mikolov et al., 2013; Devlin et al., 2018; Lan et al., 2019; Liu et al., 2019; Yang et al., 2019). While …

Content truncated to 500 words