Hints on the data for language modeling of synthetic languages with transformers

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 78–91 July 9-14, 2023 ©2023 Association for Computational Linguistics Span-Selective Linear Attention Transformers for Effective and Robust Schema-Guided Dialogue State Tracking Björn Bebensee Haejun Lee Samsung Research {b.bebensee,haejun82.lee}@samsung.com Abstract In schema-guided dialogue state tracking mod- els estimate the current state of a conversation using natural language descriptions of the ser- vice schema for generalization to unseen ser- vices. Prior generative approaches which de- code slot values sequentially do not generalize well to variations in schema, while discrimina- tive approaches separately encode history and schema and fail to account for inter-slot and intent-slot dependencies. We introduce SPLAT, a novel architecture which achieves better gen- eralization and efficiency than prior approaches by constraining outputs to a limited prediction space. At the same time, our model allows for rich attention among descriptions and history while keeping computation costs constrained by incorporating linear-time attention. We demonstrate the effectiveness of our model on the Schema-Guided Dialogue (SGD) and Mul- tiWOZ datasets. Our approach significantly improves upon existing models achieving 85.3 JGA on the SGD dataset. Further, we show in- creased robustness on the SGD-X benchmark: our model outperforms the more than 30× larger D3ST-XXL model by 5.0 points. 1 Introduction Dialogue State Tracking (DST) refers to the task of estimating and tracking the dialogue state con- sisting of the user’s current intent and set of slot- value pairs throughout the dialogue (Williams et al., 2013). Traditional approaches to DST assume a fixed ontology and learn a classifier for each slot (Chao and Lane, 2019). However, in real-world applications services can be added or removed re- quiring the model to be re-trained each time the ontology changes. Recently more flexible schema- guided approaches which take as input natural lan- guage descriptions of all available intents and slots and thus can be applied zero-shot to new services have been gaining popularity (Rastogi et al., 2020; Feng et al., 2021; Zhao et al., 2022; Gupta et al., 2022). Figure 1: Span selection for schema-guided dialogue in practice. [SLOT] encodes the semantics of the natural language description of “to_location” and is matched with the span representation of “Long Beach, CA”. Sim- ilarly [UTT] encodes the semantics of the current utter- ance and is matched with the target [INTENT] encoding. Discriminative DST models are based on ma- chine reading comprehension (MRC) methods, meaning they extract and fill in non-categorical slot values directly from the user utterances (Chao and Lane, 2019; Ruan et al., 2020; Zhang et al., 2021). We use the terms discriminative and ex- tractive interchangeably when referring to these methods. Generative DST models leverage seq2seq language models which conditioned on the dialog history and a prompt learn to sequentially generate the appropriate slot values. Prior generative meth- ods do not generalize well to variations in schema (Lee et al., 2021, 2022; Zhao et al., 2022) whereas discriminative methods separately encode history and schema and fail to account for inter-slot and intent-slot …

Content truncated to 500 words