CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4506–4515, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4506 CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text Koustuv Sinha 1,3,4, Shagun Sodhani 2,3, Jin Dong 1,3, Joelle Pineau 1,3,4 and William L. Hamilton 1,3,4 1 School of Computer Science, McGill University, Canada 2 Universit´e de Montr´eal, Canada 3 Montreal Institute of Learning Algorithms (Mila), Canada 4 Facebook AI Research (FAIR), Montreal, Canada {koustuv.sinha, sshagunsodhani, jin.dong, jpineau, wlh} @{mail.mcgill.ca, gmail.com, mail.mcgill.ca, cs.mcgill.ca, cs.mcgill.ca} Abstract The recent success of natural language under- standing (NLU) systems has been troubled by results highlighting the failure of these mod- els to generalize in a systematic and robust way. In this work, we introduce a diagnostic benchmark suite, named CLUTRR, to clarify some key issues related to the robustness and systematicity of NLU systems. Motivated by classic work on inductive logic programming, CLUTRR requires that an NLU system infer kinship relations between characters in short stories. Successful performance on this task requires both extracting relationships between entities, as well as inferring the logical rules governing these relationships. CLUTRR al- lows us to precisely measure a model’s abil- ity for systematic generalization by evaluat- ing on held-out combinations of logical rules, and it allows us to evaluate a model’s robust- ness by adding curated noise facts. Our em- pirical results highlight a substantial perfor- mance gap between state-of-the-art NLU mod- els (e.g., BERT and MAC) and a graph neu- ral network model that works directly with symbolic inputs—with the graph-based model exhibiting both stronger generalization and greater robustness. 1 Introduction Natural language understanding (NLU) systems have been extremely successful at reading compre- hension tasks, such as question answering (QA) and natural language inference (NLI). An array of existing datasets are available for these tasks. This includes datasets that test a system’s ability to ex- tract factual answers from text (Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2016; Mostafazadeh et al., 2016; Su et al., 2016), as well as datasets that emphasize commonsense inference, such as entailment between sentences (Bowman et al., 2015; Williams et al., 2018). Figure 1: CLUTRR inductive reasoning task. However, there are growing concerns regarding the ability of NLU systems—and neural networks more generally—to generalize in a systematic and robust way (Bahdanau et al., 2019; Lake and Ba- roni, 2018; Johnson et al., 2017). For instance, recent work has highlighted the brittleness of NLU systems to adversarial examples (Jia and Liang, 2017), as well as the fact that NLU models tend to exploit statistical artifacts in datasets, rather than exhibiting true reasoning and generalization capa- bilities (Gururangan et al., 2018; Kaushik and Lip- ton, 2018). These findings have also dovetailed with the recent dominance of large pre-trained lan- guage models, such as BERT, on NLU benchmarks (Devlin et al., 2018; Peters et al., 2018), which sug- gest that the primary difficulty in …

Content truncated to 500 words