How Many Data Samples is an Additional Instruction Worth?

Findings of the Association for Computational Linguistics: EACL 2023, pages 1042–1057 May 2-6, 2023 ©2023 Association for Computational Linguistics How Many Data Samples is an Additional Instruction Worth? Ravsehaj Singh Puri˚ Swaroop Mishra˚ Mihir Parmar Chitta Baral Arizona State University, Tempe, USA {rpuri8, srmishr1, mparmar3, chitta}@asu.edu Abstract Recently introduced instruction-paradigm em- powers non-expert users to leverage NLP re- sources by defining a new task in natural lan- guage. Instruction-tuned models have signif- icantly outperformed multitask learning mod- els (without instruction); however they are far from state-of-the-art task-specific models. Con- ventional approaches to improve model perfor- mance via creating datasets with large number of task instances or architectural changes in the model may not be feasible for non-expert users. However, they can write alternate instructions to represent an instruction task. Is Instruction- augmentation helpful? We augment a subset of tasks in the expanded version of NATURAL IN- STRUCTIONS with additional instructions and find that it significantly improves model perfor- mance (up to 35%), especially in the low-data regime. Our results indicate that an additional instruction can be equivalent to „200 data sam- ples on average across tasks.1 1 Introduction Large-scale benchmarks such as Imagenet (Rus- sakovsky et al., 2015), SQuAD (Rajpurkar et al., 2018) and architectural development in models such as CNNs (Amari et al., 2003) and transformers (Vaswani et al., 2017) have propelled our progress in deep learning. However, creating high-quality benchmarks by controlling its artifacts (Gururan- gan et al., 2018; Mishra et al., 2020), develop- ing new models, and training them is hard for non-expert users. Recently introduced instruction- paradigm empowers non-expert users, practition- ers, and domain experts in other fields to leverage NLP resources (Weller et al., 2020) as they now can describe their tasks in natural language without requiring to create task-specific datasets or develop- ing models2. Even though the instruction paradigm ˚Equal Contribution 1Code and dataset is available at https://github.com/ Ravsehajsinghpuri/Multi-Variant-Instructions 2Related work is presented in App. A has led to the development of models that signif- icantly outperform multitasking baselines, model performance has remained far behind the super- vised learning model trained with task-specific data (Efrat and Levy, 2020; Mishra et al., 2021b). Non-expert users can write multiple instructions per task each of which covers multiple perspec- tives spanning over a variety of linguistic features; many of these can be created automatically by re- placing certain words with their synonyms without changing the overall semantics of instruction. Can the relatively inexpensive process of instruction augmentation improve the model’s performance in the instruction-paradigm, similar to the role data-augmentation has played conventionally in machine learning (Feng et al., 2021)? Instruction- paradigm is pivotal where it is expensive or in- feasible to gather training data. How effective is instruction augmentation in low-data regimes? Multi-variant instructions (original + augmented instructions) also can help evaluate the robustness of instruction-following models to respond to vari- ant instructions. This is similar to the model ro- bustness evaluation (Jia et al., 2019) that is done by creating variant data instances. Multi-variant instruction-based setup …

Content truncated to 500 words