TestAug: A Framework for Augmenting Capability-based NLP Tests

Proceedings of the 29th International Conference on Computational Linguistics, pages 3480–3495 October 12–17, 2022. 3480 TestAug: A Framework for Augmenting Capability-based NLP Tests Guanqun Yang▲ Mirazul Haque♢ Qiaochu Song▲ Wei Yang♢ Xueqing Liu ▲ ▲Department of Computer Science, Stevens Institute of Technology ♢Department of Computer Science, The University of Texas at Dallas gyang16@stevens.edu, mirazul.haque@utdallas.edu, qsong6@stevens.com wei.yang@utdallas.edu, xliu127@stevens.edu Abstract The recently proposed capability-based NLP testing allows model developers to test the func- tional capabilities of NLP models, revealing functional failures that cannot be detected by the traditional heldout mechanism. However, existing work on capability-based testing re- quires extensive manual efforts and domain ex- pertise in creating the test cases. In this pa- per, we investigate a low-cost approach for the test case generation by leveraging the GPT-3 engine. We further propose to use a classi- fier to remove the invalid outputs from GPT-3 and expand the outputs into templates to gen- erate more test cases. Our experiments show that TestAug has three advantages over the existing work on behavioral testing: (1) Tes- tAug can find more bugs than existing work; (2) The test cases in TestAug are more di- verse; and (3) TestAug largely saves the man- ual efforts in creating the test suites. The code and data for TestAug can be found at https://github.com/guanqun-yang/testaug. 1 Introduction In recent years, natural language processing (NLP) has seen major breakthroughs in the model perfor- mances. Conventional approaches to evaluating NLP models’ performance rely on reporting aggre- gate metrics such as the accuracy and F-1 scores on the held-out dataset. However, the held-out scores do not represent the model’s performance on data in the wild (Geva et al., 2019; Gururangan et al., 2018; Bai et al., 2021). Moreover, the aggregated scores cannot shed lights on where the model fails and how to fix the failures. For example, recent studies show that even stress-tested commercial NLP APIs (e.g., Google Cloud’s Natural Language API1 and Microsoft’s Text Analytics API2) often 1https://cloud.google.com/natural-language 2https://azure.microsoft.com/en-us/services/cognitive- services/text-analytics Table 1: Example of capability-base tests on three NLP tasks: sentiment classification, paraphrase detection, and natural language inference. Task: Sentiment Classification Capability: Negation Test Description: Negative if negated positive words Input: "No one loves the food." Label: Negative Task: Paraphrase Detection Capability: Negation Test Description: Paraphrase if replacing a word with the negated antonym Input: "She is a generous person. She is not a mean person." Label: Paraphrase Task: Natural Language Inference Capability: Downward entailment Test Description: Entailment if replacing a word with its superset Input: "Some cows are brown. Some animals are brown." Label: Entailment fail on test cases of simple behaviors (Glockner et al., 2018; Ribeiro et al., 2020). To assist developers in finding behavioral fail- ures in their models, recent work has proposed a framework called CheckList for behavioral or capability-based NLP testing (Ribeiro et al., 2020; Tarunesh et al., 2021). Such tests include input and output pairs to examine the model’s performance on each linguistic capability. Table 1 shows exam- ples of capability-based tests for three NLP tasks. For …

Content truncated to 500 words