Distilling Structured Knowledge for Text-Based Relational Reasoning

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6782–6791, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 6782 Distilling Structured Knowledge for Text-Based Relational Reasoning Jin Dong 1,2, Marc-Antoine Rondeau 3, William L. Hamilton 1,2,4 1 School of Computer Science, McGill University, Canada 2 Quebec AI Institute (Mila), Canada 3 Microsoft Research, Montreal, Canada 4 Canada CIFAR AI Chair jin.dong@mail.mcgill.ca, marcantoine.rondeau@microsoft.com wlh@cs.mcgill.ca Abstract There is an increasing interest in developing text-based relational reasoning systems, which are capable of systematically reasoning about the relationships between entities mentioned in a text. However, there remains a substantial performance gap between NLP models for re- lational reasoning and models based on graph neural networks (GNNs), which have access to an underlying symbolic representation of the text. In this work, we investigate how the struc- tured knowledge of a GNN can be distilled into various NLP models in order to improve their performance. We first pre-train a GNN on a reasoning task using structured inputs and then incorporate its knowledge into an NLP model (e.g., an LSTM) via knowledge distil- lation. To overcome the difficulty of cross- modal knowledge transfer, we also employ a contrastive learning based module to align the latent representations of NLP models and the GNN. We test our approach with two state-of- the-art NLP models on 12 different inductive reasoning datasets from the CLUTRR bench- mark and obtain significant improvements. 1 Introduction The task of text-based relational reasoning—where an agent must infer and compose relations between entities based on a passage of text—has received increasing attention in natural language process- ing (NLP) (Andreas, 2019). This task has been especially prominent in the context of systematic generalization in NLP, with synthetic datasets, such as CLUTTR and SCAN, being used to probe the ability of NLP models to reason in a systematic and logical way (Lake and Baroni, 2018; Sinha et al., 2019). More generally, these investigations dovetail with the rising prominence of relational reasoning throughout machine learning and cogni- tive science (Alexander et al., 2016; Battaglia et al., 2018; Hamilton et al., 2017). However, despite the increased attention and research on text-based relational reasoning, seri- ous challenges remain. Perhaps one of the biggest challenges is the persistent gap between the per- formance that can be achieved using NLP models and the performance of structured models—such as graph neural networks (GNNs)—which perform relational reasoning based on structured or sym- bolic inputs. This gap was made particularly ev- ident in the the CLUTRR benchmark. CLUTRR includes relational reasoning problems that can be posed both in textual or symbolic form, and preliminary investigations using CLUTRR show that GNN-based models—which leverage the struc- tured symbolic input—are able to achieve higher accuracy, better generalization, and are more robust than purely text-based systems (Sinha et al., 2019). In this work, we investigate one potential av- enue to close this gap. We design an approach to distill the structured knowledge learned by a GNN— which has access to the underlying symbolic rep- resentation of …

Content truncated to 500 words