Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective

Findings of the Association for Computational Linguistics: ACL 2022, pages 2355 - 2366 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Systematicity, Compositionality and Transitivity of Deep NLP Models: a Metamorphic Testing Perspective Edoardo Manino University of Manchester United Kingdom Julia Rozanova University of Manchester United Kingdom Danilo Carvalho University of Manchester United Kingdom André Freitas University of Manchester, United Kingdom Idiap Research Institute, Switzerland Lucas Cordeiro University of Manchester United Kingdom Abstract Metamorphic testing has recently been used to check the safety of neural NLP models. Its main advantage is that it does not rely on a ground truth to generate test cases. However, existing studies are mostly concerned with robustness-like metamorphic relations, limit- ing the scope of linguistic properties they can test. We propose three new classes of meta- morphic relations, which address the prop- erties of systematicity, compositionality and transitivity. Unlike robustness, our relations are defined over multiple source inputs, thus increasing the number of test cases that we can produce by a polynomial factor. With them, we test the internal consistency of state-of-the- art NLP models, and show that they do not al- ways behave according to their expected lin- guistic properties. Lastly, we introduce a novel graphical notation that efficiently summarises the inner structure of metamorphic relations. 1 Introduction Many recent advances in neural models for NLP have been driven by the ability to learn from unla- beled data (Devlin et al., 2019; Liu et al., 2019b). This approach allows for training the models on large-scale corpora without the costly process of annotating them. As a result, the accuracy and com- plexity of state-of-the-art neural models for NLP have increased (Brown et al., 2020). This trend towards unlabeled data does not have a counterpart in testing NLP models. Instead, both in-distribution testing and out-of-distribution test- ing (Yin et al., 2019; Teney et al., 2020) rely on comparing the model’s predictions to the ground truth. Similarly, attempts at probing the internal computation of large NLP models use supervised classifiers as a diagnostic tool (Ettinger et al., 2016; Belinkov et al., 2017). In general, such extreme reliance on ground- truth data limits the quantity of test cases we can produce, which is a known problem in the soft- ware testing community (Barr et al., 2015). In this regard, a promising solution is metamorphic test- ing (Chen et al., 2018). Under this paradigm, we test the internal consistency of an NLP model by checking whether it satisfies a necessary relation of its inputs and outputs (Ribeiro et al., 2020). Conse- quently, metamorphic testing relies on our ability to formally express our expectations on the behaviour of an NLP model. Still, most of the metamorphic relations pro- posed in the literature target the same type of be- haviour, as we show in this paper. Indeed, the majority of them are robustness relations, which require that the output of an NLP model remains stable in the face of small input perturbations (As- pillaga et al., 2020). These perturbations may in- …

Content truncated to 500 words