“No, They Did Not”: Dialogue Response Dynamics in Pre-trained Language Models

Proceedings of the 29th International Conference on Computational Linguistics, pages 863–874 October 12–17, 2022. 863 “No, they did not”: Dialogue response dynamics in pre-trained language models Sanghee J. Kim1, Lang Yu2, Allyson Ettinger1 1Department of Linguistics, University of Chicago 2Meta {sangheekim,aettinger}@uchicago.edu, langyu@fb.com Abstract A critical component of competence in lan- guage is being able to identify relevant com- ponents of an utterance and reply appropriately. In this paper we examine the extent of such dialogue response sensitivity in pre-trained lan- guage models, conducting a series of experi- ments with a particular focus on sensitivity to dynamics involving phenomena of at-issueness and ellipsis. We find that models show clear sensitivity to a distinctive role of embedded clauses, and a general preference for responses that target main clause content of prior utter- ances. However, the results indicate mixed and generally weak trends with respect to capturing the full range of dynamics involved in target- ing at-issue versus not-at-issue content. Addi- tionally, models show fundamental limitations in grasp of the dynamics governing ellipsis, and response selections show clear interference from superficial factors that outweigh the influ- ence of principled discourse constraints. 1 Introduction Competence in language involves understanding complex principles governing relevance of previous content and dynamics of referring back to that con- tent. Certain parts of an utterance are more central and more likely to receive a response than others, and the pragmatic and grammatical rules governing responses in dialogue interact with the nature of the content being responded to. Humans are highly sensitive to these distinctions, and we can expect these sensitivities to be critical for robust models in NLP, and especially for dialogue. Here we examine sensitivity to these dialogue response dynamics in pre-trained language mod- els (PLMs). PLMs are now used as foundation for nearly every downstream NLP task, including dialogue applications (e.g., Upadhye et al., 2020; Koto et al., 2021). The impressive downstream performance enabled by these models has raised important questions about what types of linguistic competence are being learned during pre-training— and though there is a growing body of work answer- ing aspects of this question, topics of pragmatic and dialogue competence have been relatively under- studied. In this paper we focus on addressing this gap, and in particular on understanding the extent to which PLMs develop sensitivity to dynamics governing responses in dialogue. Though these PLMs are not trained to engage in dialogue per se, they can be expected to encounter dialogue dur- ing training (in, for instance, novels), so it is not unreasonable to expect that they may learn about such dialogue dynamics along with other linguis- tic competences. The strength of these models’ sensitivity to such dynamics has important implica- tions for robustness in dialogue applications, since a strong grasp of dialogue dynamics in standard PLMs stands to reduce fine-tuning needs and en- able more robust downstream behaviors. We begin with the notion of at-issueness. A com- ponent of an utterance is considered at-issue if it is part of the “main point” of …

Content truncated to 500 words