Dialogue Coherence Assessment Without Explicit Dialogue Act Labels

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1439–1450 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 1439 Dialogue Coherence Assessment Without Explicit Dialogue Act Labels Mohsen Mesgar Sebastian B¨ucker Ubiquitous Knowledge Processing Lab (UKP) Technische Universit¨at Darmstadt (TUDa) {mesgar,buecker,gurevych}@ukp.tu-darmstadt.de Iryna Gurevych Abstract Recent dialogue coherence models use the coherence features designed for monologue texts, e.g. nominal entities, to represent utter- ances and then explicitly augment them with dialogue-relevant features, e.g., dialogue act labels. It indicates two drawbacks, (a) seman- tics of utterances is limited to entity mentions, and (b) the performance of coherence mod- els strongly relies on the quality of the input dialogue act labels. We address these issues by introducing a novel approach to dialogue coherence assessment. We use dialogue act prediction as an auxiliary task in a multi-task learning scenario to obtain informative utter- ance representations for coherence assessment. Our approach alleviates the need for explicit dialogue act labels during evaluation. The re- sults of our experiments show that our model substantially (more than 20 accuracy points) outperforms its strong competitors on the Dai- lyDialogue corpus, and performs on par with them on the SwitchBoard corpus for ranking dialogues concerning their coherence. We re- lease our source code1. 1 Introduction Considering rapid progresses in developing open-domain dialogue agents (Serban et al., 2016; Ghazvininejad et al., 2018; Dinan et al., 2019; Li et al., 2019), the need for models that compare these agents in various dialogue aspects becomes extremely important (Liu et al., 2016; Dinan et al., 2019). Most available methods for dialogue evalua- tion rely on word-overlap metrics, e.g. BLEU, and manually collected human feedback. The former does not strongly correlate with human judgments (Liu et al., 2016), and the latter is time-consuming and subjective. A fundamental aspect of dialogue is coherence – what discriminates a high-quality 1https://github.com/UKPLab/ acl2020-dialogue-coherence-assessment utterances shared utterance encoder DAP model DA labels DiCoh model coherence score Figure 1: A high-level view of our multi-task learning approach for dialogue coherence modeling. dialogue from a random sequence of dialogue ut- terances (Halliday and Hasan, 1976; Grosz and Sidner, 1986; Byron and Stent, 1998). Dialogue coherence deals with semantic relations between utterances considering their dialogue acts (Perrault and Allen, 1978; Cervone et al., 2018). A Dialogue Act (henceforth DA) gives a meaning to an utterance in a dialogue at the level of “illocu- tionary force”, and therefore, constitutes the basic unit of communication (Searle, 1969; Raheja and Tetreault, 2019). A DA captures what a speaker’s intention is of saying an utterance without regard to the actual content of the utterance. For example, a DA may indicate whether the intention of stating an utterance is to ask a question or to state a piece of information. Recent approaches to dialogue coherence model- ing use the coherence features designed for mono- logue texts, e.g. entity transitions (Barzilay and Lapata, 2005), and augment them with dialogue- relevant features, e.g., DA labels (Cervone et al., 2018). These DA labels are provided …

Content truncated to 500 words