How Reliable are Model Diagnostics?

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1778–1785 August 1–6, 2021. ©2021 Association for Computational Linguistics 1778 How Reliable are Model Diagnostics? Vamsi Aribandi∗ Google Research aribandi@google.com Yi Tay Google Research yitay@google.com Donald Metzler Google Research metzler@google.com Abstract In the pursuit of a deeper understanding of a model’s behaviour, there is recent impetus for developing suites of probes aimed at di- agnosing models beyond simple metrics like accuracy or BLEU. This paper takes a step back and asks an important and timely ques- tion: how reliable are these diagnostics in pro- viding insight into models and training setups? We critically examine three recent diagnostic tests for pre-trained language models, and find that likelihood-based and representation-based model diagnostics are not yet as reliable as pre- viously assumed. Based on our empirical find- ings, we also formulate recommendations for practitioners and researchers. 1 Introduction Contemporary statistical models based on deep learning have made incredible progress towards solving complex language tasks (Radford et al., 2019; Devlin et al., 2019; Raffel et al., 2020). These models generally trade off the interpretability and simplicity of traditional models for powerful parameterizations and inductive biases, enabling their impressive performance. However, their en- try into critical fields such as medicine, the justice system, and social media moderation often makes this trade-off a costly one. Consequently, there has been surging interest in the development of tools and suites for diagnosing and better understand- ing model behaviour, and gaining insight into what patterns and phenomena they have learned (§4.1). Ideally, these diagnostics would not only help practitioners understand the failure modes and ca- pabilities of large contemporary models, but also enable them to improve their models based on the diagnostics. To this end, we believe that model diagnostics are essential for making meaningful progress in natural language processing. ∗Google AI Resident Model diagnostics generally probe a model for specific learned qualities (§4.1). These may be a positive qualities (e.g., whether a model has ac- quired syntactic knowledge) or potentially problem- atic qualities (e.g., biases and stereotypes. These probes can be used to identify certain phenomena that can be used to further improve models. Given the potential impact that model diagnos- tics can have for practitioners and the research community’s fundamental understanding of con- temporary models, this paper asks the important and inevitable question of whether these probes are actually reliable and robust, and to what extent they are. These diagnostics’ explicit nature as a tool for understanding also imposes a greater bar for robustness, as inconsistencies may mislead and result in compounding errors. Our findings demonstrate that model diagnos- tics can be unreliable on multiple fronts. To illus- trate our point, we select three diagnostics tasks — StereoSet (Nadeem et al., 2020), CrowS-Pairs (Nangia et al., 2020), and SEATs (May et al., 2019) to base our empirical evaluation on. Overall, we find that likelihood-based and representation- based diagnostics measured multiple times on the same training setup can result in wildly dif- ferent findings. Specifically, a substantial variance …

Content truncated to 500 words