Do dialogue representations align with perception? An empirical study

Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2696–2713 May 2-6, 2023 ©2023 Association for Computational Linguistics Do dialogue representations align with perception? An empirical study Sarenne Wallbridge and Peter Bell and Catherine Lai Center for Speech Technology Research University of Edinburgh, United Kingdom {s1301730, peter.bell, c.lai}@ed.ac.uk Abstract There has been a surge of interest regarding the alignment of large-scale language mod- els with human language comprehension be- haviour. The majority of this research inves- tigates comprehension behaviours from read- ing isolated, written sentences. We propose studying the perception of dialogue, focusing on an intrinsic form of language use: spoken conversations. Using the task of predicting up- coming dialogue turns, we ask whether turn plausibility scores produced by state-of-the-art language models correlate with human judge- ments. We find a strong correlation for some but not all models: masked language mod- els produce stronger correlations than auto- regressive models. In doing so, we quantify human performance on the response selection task for open-domain spoken conversation. To the best of our knowledge, this is the first such quantification. We find that response selection performance can be used as a coarse proxy for the strength of correlation with human judge- ments, however humans and models make dif- ferent response selection mistakes. The model which produces the strongest correlation also outperforms human response selection perfor- mance. Through ablation studies, we show that pre-trained language models provide a useful basis for turn representations; however, fine- grained contextualisation, inclusion of dialogue structure information, and fine-tuning towards response selection all boost response selection accuracy by over 30 absolute points. 1 Introduction Human language processing has intrigued re- searchers from numerous fields for centuries (Herder, 1772). The relatively recent convergence of philosophy, psycholinguistics, and information theory has produced valuable theories of language production and comprehension by framing people as predictive processors (Christiansen and Chater, 2016; Levy, 2008; Hale, 2001). In particular, Sur- prisal Theory posits that comprehension effort is directly related to the predictability of a linguistic unit in its context (i.e., its surprisal) (Hale, 2001), and the Smooth Signal Redundancy hypothesis (Aylett and Turk, 2004) (and related theories includ- ing Uniform Information Density (Fenk and Fenk, 1980; Levy and Jaeger, 2006) and Entropy Rate Constancy (Genzel and Charniak, 2002)) demon- strates that we produce linguistic signals that tend towards uniform distributions of information, or constant predictability. Such theories rely on the conditional probability of observing a linguistic unit in a particular context. Traditionally, estimates of these probabilities were obtained through statistical models such as n-grams or PCFGs (Smith and Levy, 2013; Hale, 2001). By comparison, large scale language models (LLMs) like Transformers allow much greater degrees of context to be integrated into probability estimates (Vaswani et al., 2017). LLMs have enabled massive progress across the continuum of NLP tasks (Wang et al., 2019; Hu et al., 2020). As such, a recent field of research has investigated the application of LLMs for producing estimates of human surprisal. Language model quality …

Content truncated to 500 words