Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking

Proceedings of the 11th Joint Conference on Lexical and Computational Semantics, pages 101 - 122 July 14-15, 2022 ©2022 Association for Computational Linguistics Dyna-bAbI: unlocking bAbI’s potential with dynamic synthetic benchmarking Ronen Tamari†∗ Kyle Richardson⋆ Noam Kahlon† Aviad Sar-Shalom△ Nelson F. Liu⋄ Reut Tsarfaty⋆‡ Dafna Shahaf† †The Hebrew University of Jerusalem ⋆Allen Institute for AI ‡Bar-Ilan University △Tel-Aviv University ⋄Stanford University {ronent,dshahaf}@cs.huji.ac.il, {reutt,kyler}@allenai.org Abstract While neural language models often perform surprisingly well on natural language understanding (NLU) tasks, their strengths and limitations remain poorly understood. Controlled synthetic tasks are thus an increasingly important resource for diagnosing model behavior. In this work we focus on story understanding, a core competency for NLU systems. However, the main synthetic resource for story understanding, the bAbI benchmark, lacks such a systematic mechanism for controllable task generation. We develop Dyna-bAbI, a dynamic framework providing fine-grained control over task generation in bAbI. We demonstrate our ideas by constructing three new tasks requiring compositional generalization, an important evaluation setting absent from the original benchmark. We tested both special-purpose models developed for bAbI as well as state-of-the-art pre-trained methods, and found that while both approaches solve the original tasks (>99% accuracy), neither approach succeeded in the compositional generalization setting, indicating the limitations of the original training data. We explored ways to augment the original data, and found that though diversifying training data was far more useful than simply increasing dataset size, it was still insufficient for driving robust compositional generalization (with <70% accuracy for complex compositions). Our results underscore the importance of highly controllable task generators for creating robust NLU systems through a virtuous cycle of model and data development.1 1 Introduction Considerable progress has been made recently in natural language understanding (NLU), driven largely by advances in model pre-training (Devlin ∗ Work begun during an internship at the Allen Institute. 1 Data and code available at https:// dyna-babi-project.github.io/. Figure 1: (a) Low task configurability leads to static datasets, benchmark saturation & unreliable model development. (b) We propose a dynamic benchmarking approach; developing models and tasks in a tight feedback loop using (c) Dyna-bAbI task generator. Dyna-bAbI provides fine-grained control over task structure, composition and difficulty, yielding challenging new test sets exposing limitations of state- of-the-art models. et al., 2019; Raffel et al., 2020) and the development of large-scale NLU benchmarks across a wide range of tasks (Wang et al., 2018, 2019; Liang et al., 2020). Such successes, however, have coincided with the discovery of various shortcomings in existing human curated datasets, largely related to annotation artifacts (Gururangan 101 et al., 2018), or systematic biases that create shortcuts that can inflate model performance and harm generalization. In order to overcome these issues, two avenues of research have recently gained traction: 1) development of dynamic benchmarks (Potts et al., 2021; Kiela et al., 2021) where, in contrast to conventional static benchmarks, evaluation and data collection are conducted interactively with humans and models in a rapidly evolving feedback loop and; 2) renewed interest in synthetic benchmarks (Lake and Baroni, 2018; …

Content truncated to 500 words