Soft Alignment Objectives for Robust Adaptation of Language Generation

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 8837–8853 July 9-14, 2023 ©2023 Association for Computational Linguistics Soft Alignment Objectives for Robust Adaptation of Language Generation Michal Štefánik♣∗ and Marek Kadlčík♣ and Petr Sojka♣ ♣Faculty of Informatics, Masaryk University, Czech Republic Abstract Domain adaptation allows generative language models to address specific flaws caused by the domain shift of their application. However, the traditional adaptation by further training on in- domain data rapidly weakens the model’s ability to generalize to other domains, making the open- ended deployments of the adapted models prone to errors. This work introduces novel training objectives built upon a semantic similarity of the predicted tokens to the reference. Our results show that (1) avoiding the common assumption of a single correct prediction by con- structing the training target from tokens’ seman- tic similarity can largely mitigate catastrophic forgetting of adaptation, while (2) preserving the adaptation in-domain quality, (3) with negli- gible additions to compute costs. In the broader context, the objectives grounded in a contin- uous token similarity pioneer the exploration of the middle ground between the efficient but naïve exact-match token-level objectives and expressive but computationally- and resource- intensive sequential objectives. 1 Introduction Large language models (LLMs) based on instances of encoder-decoder architecture (Neyshabur et al., 2015) provide a strong standard for generative appli- cations of NLP, such as summarization or machine translation, mainly thanks to their outstanding abil- ity to fluently model language. These models might face issues with adequacy of the generated text (Us- taszewski, 2019) when applied in data domain(s) different from the training domain, but such errors can be partially mitigated using domain adapta- tion (Saunders, 2021). Identically to the pre-training phase, the adap- tation is commonly carried out using Maximum Likelihood Estimation (MLE) objective with teacher forcing (Bahdanau et al., 2015). The popularity of *Corresponding author: stefanik.m@mail.muni.cz src: 多伦多很好。 ref: Toronto is ___ y: nice stunning lovely dark y: nice stunning lovely dark MLE (previous work) *Align objectives (ours) Predicted Model A y ref Target Figure 1: Soft alignment objectives (*Align) replace the single-truth assumption of Maximum Likelihood Estimation (MLE) objective by constructing target distri- bution using Alignment A based on the mutual similarity of token representations. We show that learning to model ambiguity in prediction can largely mitigate the loss of generalization in adaptation. this approach can be rightfully attributed to its out- standing data and computing efficiency. However, model adaptation using MLE notoriously comes for a price of over-specialization to the target domain, also referred to as catastrophic forgetting (Good- fellow et al., 2014), characterized by a continuous decay of model performance on the inputs from the other domains than the adaptation domain. We hypothesize that catastrophic forgetting might be related to MLE’s naïve single-truth assumption, penalizing models’ uncertainty over the possibly valid predictions, such as the synonyms. In domain adaptation, a repeated penalization of possibly valid tokens that are uncommon in the adapted domain might drive the model to …

Content truncated to 500 words