Space Efficient Context Encoding for Non-Task-Oriented Dialogue Generation with Graph Attention Transformer

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 7028–7041 August 1–6, 2021. ©2021 Association for Computational Linguistics 7028 Space Efficient Context Encoding for Non-Task-Oriented Dialogue Generation with Graph Attention Transformer Fabian Galetzka1,2,∗, Jewgeni Rose1,3,∗,†, David Schlangen2 and Jens Lehmann3,4 1Volkswagen Innovation Center, Wolfsburg, Germany 2Computational Linguistics, University of Potsdam, Germany 3University of Bonn, Germany 4Fraunhofer IAIS, Dresden, Germany {jewgeni.rose,fabian.galetzka}@volkswagen.de jens.lehmann@iais.fraunhofer.de david.schlangen@uni-potsdam.de Abstract To improve the coherence and knowledge re- trieval capabilities of non-task-oriented dia- logue systems, recent Transformer-based mod- els aim to integrate fixed background context. This often comes in the form of knowledge graphs, and the integration is done by cre- ating pseudo utterances through paraphrasing knowledge triples, added into the accumulated dialogue context. However, the context length is fixed in these architectures, which restricts how much background or dialogue context can be kept. In this work, we propose a more con- cise encoding for background context struc- tured in the form of knowledge graphs, by ex- pressing the graph connections through restric- tions on the attention weights. The results of our human evaluation show that this encod- ing reduces space requirements without nega- tive effects on the precision of reproduction of knowledge and perceived consistency. Further, models trained with our proposed context en- coding generate dialogues that are judged to be more comprehensive and interesting. 1 Introduction Building on the idea of attention-based seq2seq models (Vaswani et al., 2017), recent language models such as BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019) enable neural conver- sational models to generate responses that appear human-like and engaging (Yu et al., 2019). A closer look, however, reveals that the lack of long-term memory to represent consistent (world) knowledge and personality over multiple speaker turns can lead to incoherent content being generated (Li et al., 2016; Serban et al., 2017). Initiated by the Con- versational Intelligence Challenge (Burtsev et al., 2018; Dinan et al., 2020), the research focus there- fore shifted towards knowledge-grounded dialogue ∗The first two authors contributed equally to this paper. †Corresponding author generation, resulting in first promising approaches using Transformer-based architectures (Dinan et al., 2019; Ghazvininejad et al., 2018; Galetzka et al., 2020). The basic idea of these approaches is to provide the required background knowledge together with the current dialogue context when decoding the next system utterance. As the underlying language model’s input sequence length is limited – for in- stance, to 1024 tokens in the case of GTP-2 – the presentation of the background knowledge to the model highly impacts the amount of context infor- mation that can be fed into a Transformer network. In these earlier attempts, the knowledge was para- phrased into pseudo-utterances, on a par with the utterances from the dialogue history. In this paper, we show that a structured knowledge representa- tion offers advantages over unstructured text: facts and complex relationships between different enti- ties can be encoded concisely without performance drop in key indicators, …

Content truncated to 500 words