Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking

Findings of the Association for Computational Linguistics: EACL 2023, pages 1551–1564 May 2-6, 2023 ©2023 Association for Computational Linguistics Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking Derek Chen, Kun Qian, Zhou Yu Dialogue NLP Lab Columbia University {dc3761, kq2157, zy2461}@columbia.edu Abstract Prompt-based methods with large pre-trained language models (PLMs) have shown impres- sive unaided performance across many NLP tasks. These models improve even further with the addition of a few labeled in-context ex- emplars to guide output generation. However, for more complex tasks such as dialogue state tracking (DST), designing prompts that reliably convey the desired intent is nontrivial, leading to unstable results. Furthermore, building in- context exemplars for dialogue tasks is difficult because conversational contexts are long while model input lengths are relatively short. To overcome these issues we first adapt a meta- learning scheme to the dialogue domain which stabilizes the ability of the model to perform well under various prompts. We additionally design a novel training method to improve upon vanilla retrieval mechanisms to find ideal in-context examples. Finally, we introduce a saliency model to limit dialogue text length, al- lowing us to include more exemplars per query. In effect, we are able to achieve highly compet- itive results for few-shot DST on MultiWOZ. 1 Introduction Tremendous gains have been made on dialogue state tracking (DST) using large pre-trained lan- guage models (PLMs) (Hosseini-Asl et al., 2020; Peng et al., 2021), Fine-tuning such systems though require significant amounts of data, which in turn require substantial effort to collect. Recently, prompting has emerged as a technique for achiev- ing strong performance in a less resource inten- sive manner (Schick and Schütze, 2021; Liu et al., 2021). Even better performance is possible with in-context exemplars providing a pattern for the model to follow (Brown et al., 2020). Ideally, we should be able to apply these concepts to complex tasks like DST, but results so far have been lim- ited (Madotto et al., 2021). Exemplars Prompt Query Exemplar Context: I would like a train to Cambridge arriving at 14:00 Exemplar Prompt: Train destination is Exemplar Target: cambridge Naive: Area of the restaurant Question: Where is the desired restaurant location? Statement: The preferred restaurant area is Previous Context: <customer> Can you help me find somewhere to eat? <agent> Sure, what type of food would you prefer? <customer> Korean food sounds pretty good right now. Current Turn: <agent> We have many places that fit that criteria. Do you have a specific area in mind? <customer> In the north. Dialogue Query Possible Prompts T5 Target Value: in the north > Calculate Loss Predicted Value: north Meta-Training Inference No Gradient Update! Exemplar Context: I would like a train to Cambridge arriving at 14:00 Exemplar Prompt: Train destination is Exemplar Target: cambridge Exemplar Context: I would like a train to Cambridge arriving at 14:00. Exemplar Prompt: Train destination is Exemplar Target: cambridge In-Context Exemplars Figure 1: Our system squeezes multiple in-context ex- emplars, dialogue query with conversational context, and …

Content truncated to 500 words