Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion

Findings of the Association for Computational Linguistics: EACL 2023, pages 1010–1022 May 2-6, 2023 ©2023 Association for Computational Linguistics Improving Prediction Backward-Compatiblility in NLP Model Upgrade with Gated Fusion Yi-An Lai♠ Elman Mansimov♠ Yuqing Xie♡,∗ Yi Zhang♠ ♠AWS AI Labs ♡University of Waterloo {yianl,mansimov,yizhngn}@amazon.com yuqing.xie@uwaterloo.ca Abstract When upgrading neural models to a newer ver- sion, new errors that were not encountered in the legacy version can be introduced, known as regression1errors. This inconsistent behavior during model upgrade often outweighs the ben- efits of accuracy gain and hinders the adoption of new models. To mitigate regression errors from model upgrade, distillation and ensemble have proven to be viable solutions without sig- nificant compromise in performance. Despite the progress, these approaches attained an in- cremental reduction in regression which is still far from achieving backward-compatible model upgrade. In this work, we propose a novel method, Gated Fusion, that promotes backward compatibility via learning to mix predictions between old and new models. Empirical results on two distinct model upgrade scenarios show that our method reduces the number of regres- sion errors by 62% on average, outperforming the strongest baseline by an average of 25%. 1 Introduction In order to achieve a smooth continuous improve- ment of NLP applications, it is critical to guarantee consistent operation of the system after an upgrade. New errors introduced during the model upgrade interfere with the existing user experience and are considered to be a regression in the quality. Due to the difficulty of modularizing or explaining the be- havior of deep neural networks, traditional software regression tests are inapplicable to neural based systems. The cost of arduous error analysis and model patching often exceeds the benefits of model upgrades. Developing methods that ensure back- ward compatibility during model upgrades without compromise in performance becomes a valuable re- search direction (Yan et al., 2021; Xie et al., 2021; Träuble et al., 2021; Cai et al., 2022). ∗Work done during author’s internship at AWS AI Labs. 1Within this work, regression denotes performance degra- dation in software systems, instead of the statistical technique for estimating relationships among variables. BERT Correction (2/9) Old Incorrect (2/9) Upgrade Regression (1/9) ELECTRA Figure 1: Illustration of regression errors when up- grading from BERT (Devlin et al., 2019) to ELEC- TRA (Clark et al., 2020) for classification. Red circles and green squares denote examples of different classes. Dashed lines represent decision boundaries. The prediction backward-compatible model up- grade problem aims to improve consistency of cor- rect classification predictions between legacy and upgraded models without accuracy loss. Yan et al. (2021) first studied backward compatibility during model upgrade on image classification tasks. They proposed to enforce the positive congruence of the new model with the old one by applying a knowl- edge distillation objective (Hinton et al., 2015) ob- jective with re-weighting of training samples. Later, Xie et al. (2021) extended the work of Yan et al. (2021) by investigating the backward compatibility in NLP classification tasks. They found that their proposed distillation-based approach can help …

Content truncated to 500 words