Exploring Textual and Speech information in Dialogue Act Classification with Speaker Domain Adaptation

Exploring Textual and Speech information in Dialogue Act Classification with Speaker Domain Adaptation Xuanli He∗ Quan Hung Tran∗ William Havard Monash University Monash University Univ. Grenoble Alpes xuanli.he1@monash.edu hung.tran@monash.edu william.havard@gmail.com Laurent Besacier Ingrid Zukerman Gholamreza Haffari Univ. Grenoble Alpes Monash University Monash University laurent.besacier@imag.fr ingrid.zukerman@monash.edu gholamreza.haffari@monash.edu Abstract In spite of the recent success of Dia- logue Act (DA) classification, the major- ity of prior works focus on text-based clas- sification with oracle transcriptions, i.e. human transcriptions, instead of Auto- matic Speech Recognition (ASR)’s tran- scriptions. Moreover, the performance of this classification task, because of speaker domain shift, may deteriorate. In this pa- per, we explore the effectiveness of using both acoustic and textual signals, either or- acle or ASR transcriptions, and investigate speaker domain adaptation for DA classi- fication. Our multimodal model proves to be superior to the unimodal models, par- ticularly when the oracle transcriptions are not available. We also propose an effec- tive method for speaker domain adapta- tion, which achieves competitive results. 1 Introduction Dialogue Act (DA) classification is a sequence- labelling task, mapping a sequence of utterances to their corresponding DAs. Since DA classification plays an important role in understanding sponta- neous dialogue (Stolcke et al., 2000), numerous techniques have been proposed to capture the se- mantic correlation between utterances and DAs. Earlier on, statistical techniques such as Hid- den Markov Models (HMMs) were widely used to recognise DAs (Stolcke et al., 2000; Julia et al., 2010). Recently, due to the enormous success of neural networks in sequence labeling/transduction tasks (Sutskever et al., 2014; Bahdanau et al., 2014; Popov, 2016), several recurrent neural net- work (RNN) based architectures have been pro- posed to conduct DA classification, resulting in ∗Equal contribution promising outcomes (Ji et al., 2016; Shen and Lee, 2016; Tran et al., 2017a). Despite the success of previous work in DA classification, there are still several fundamental issues. Firstly, most of the previous works rely on transcriptions (Ji et al., 2016; Shen and Lee, 2016; Tran et al., 2017a). Fewer of these focus on combining speech and textual signals (Julia et al., 2010), and even then, the textual signals in these works utilise the oracle transcriptions. We argue that in the context of a spoken dialog system, or- acle transcriptions of utterances are usually not available, i.e. the agent does not have access to the human transcriptions. Speech and textual data complement each other, especially when textual data is from ASR systems rather than oracle tran- scripts. Furthermore, domain adaptation in text or speech-based DA classification is relatively under- investigated. As shown in our experiments, DA classification models perform much worse when they are applied to new speakers. In this paper, we explore the effectiveness of us- ing both acoustic and textual signals, and investi- gate speaker domain adaptation for DA classifica- tion. We present a multimodal model to combine text and speech signals, which proves to be supe- rior to the unimodal models, particularly when the oracle transcriptions are not available. Moreover, we propose …

Content truncated to 500 words