Hierarchical Pre-training for Sequence Labelling in Spoken Dialog

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2636–2648 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2636 Hierarchical Pre-training for Sequence Labelling in Spoken Dialog Emile Chapuis1⇤, Pierre Colombo1,2⇤, Matteo Manica3 Matthieu Labeau1, Chloe Clavel1 1LTCI, Telecom Paris, Institut Polytechnique de Paris, 2IBM GBS France, 3IBM Research Zurich 1firstname.lastname@telecom-paris.fr, 3tte@zurich.ibm.com Abstract Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. In this work, we propose a new approach to learn generic representations adapted to spoken dialog, which we evaluate on a new benchmark we call Sequence labellIng evaLuatIon benChmark fOr spoken laN- guagE benchmark (SILICONE). SILICONE is model-agnostic and contains 10 di↵er- ent datasets of various sizes. We obtain our representations with a hierarchical en- coder based on transformer architectures, for which we extend two well-known pre- training objectives. Pre-training is per- formed on OpenSubtitles: a large corpus of spoken dialog containing over 2.3 billion of tokens. We demonstrate how hierarchical encoders achieve competitive results with consistently fewer parameters compared to state-of-the-art models and we show their importance for both pre-training and fine- tuning. 1 Introduction The identification of both Dialog Acts (DA) and Emotion/Sentiment (E/S) in spoken lan- guage is an important step toward improving model performances on spontaneous dialogue task. Especially, it is essential to avoid the generic response problem, i.e., having an au- tomatic dialog system generate an unspecific response — that can be an answer to a very large number of user utterances (Yi et al., 2019; Colombo et al., 2019). DA and emo- tion identification (Witon et al., 2018; Jalalzai et al., 2020) are done through sequence la- belling systems that are usually trained on large corpora (with over 100k labelled utter- ances) such as Switchboard (Godfrey et al., ⇤Equal contribution 1992), MRDA (Shriberg et al., 2004) or Daily Dialog Act (Li et al., 2017). Even though large corpora enable learning complex models from scratch (e.g., seq2seq (Colombo et al., 2020)), those models are very specific to the labelling scheme employed. Adapting them to di↵erent sets of emotions or dialog acts would require more annotated data. Generic representations (Mikolov et al., 2013; Pennington et al., 2014; Peters et al., 2018; De- vlin et al., 2018; Yang et al., 2019; Liu et al., 2019) have been shown to be an e↵ective way to adapt models across di↵erent sets of labels. Those representations are usually trained on large written corpora such as OSCAR (Su´arez et al., 2019), Book Corpus (Zhu et al., 2015) or Wikipedia (Denoyer and Gallinari, 2006). Although achieving state-of-the-art (SOTA) results on written benchmarks (Wang et al., 2018), they are not tailored to spoken dialog (SD). Indeed, Tran et al. (2019) have suggested that training a parser on conversational speech data can improve results, due to the discrep- ancy between spoken and written language (e.g., disfluencies (Stolcke and Shriberg, 1996), fillers (Shriberg, 1999; Dinkar et al., 2020), dif- ferent data distribution). Furthermore, captur- ing discourse-level features, which distinguish dialog from other …

Content truncated to 500 words