Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6589–6602 August 1–6, 2021. ©2021 Association for Computational Linguistics 6589 Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates Yuqing Xie∗1, Yi-An Lai2, Yuanjun Xiong2, Yi Zhang2, Stefano Soatto2 1University of Waterloo 2Amazon AWS AI yuqing.xie@uwaterloo.ca {yianl,yuanjx,yizhngn,soattos}@amazon.com Abstract Behavior of deep neural networks can be inconsistent between different versions. Re- gressions1during model update are a common cause of concern that often over-weigh the benefits in accuracy or efficiency gain. This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. Using negative flip rate as regres- sion measure, we show that regression has a prevalent presence across tasks in the GLUE benchmark. We formulate the regression-free model updates into a constrained optimiza- tion problem, and further reduce it into a relaxed form which can be approximately op- timized through knowledge distillation train- ing method. We empirically analyze how model ensemble reduces regression. Finally, we conduct CHECKLIST behavioral testing to understand the distribution of regressions across linguistic phenomena, and the efficacy of ensemble and distillation methods. 1 Introduction Regression-free model update is a desirable system property which guarantees interoperability of a new system with a legacy version, also known as backward compatibility. Regression occurs when the newly updated system stops functioning as intended. As advances in deep learning spark industrial applications in AI areas such as natural language processing, the long-term maintenance of such systems is becoming ever more challenging. While models with complex neural architectures and huge parameter space continue to reach higher accuracy, the lack of interpretability and functional decomposibility in these models make it infeasible to apply traditional software regression testing ∗ Work done while at Amazon AWS AI. 1Here regression refers to bugs in software testing instead of the statistical estimation method. Negative No Flip Positive Flip Negative Flip Positive No Flip I can become more lovely. ≠ I can become less lovely. I can become more brave. = I can become more courageous. Alice is proposing to Bob. ≠ Bob is proposing to Alice. Sare took the castle. ≠ Sara was taken by the castle. New correct New wrong Old correct Old wrong New: ≠ Old: ≠ New: ≠ Old: = Old: = Old: = New: = New: ≠ Figure 1: Prediction flip scenarios on NLP classifi- cation tasks when updating from old to new models. Examples from paraphrase classification task. methods such as unit tests. As result, validating and mitigating regressions during model update is often a long and painful engineering process, which often over-shadows the benefits of a new model. The model regression issue in deep learning first comes into sight in Shen et al. (2020), where they inspect compatible representation learning for image retrieval. Yan et al. (2020) proposed the positive-congruent training (PCT) for image classification that minimizes prediction errors and model regression at the same time. To our best …

Content truncated to 500 words