Augmenting Neural Networks with First-order Logic

Augmenting Neural Networks with First-order Logic Tao Li University of Utah tli@cs.utah.edu Vivek Srikumar University of Utah svivek@cs.utah.edu Abstract Today, the dominant paradigm for training neural networks involves minimizing task loss on a large dataset. Using world knowledge to inform a model, and yet retain the ability to perform end-to-end training remains an open question. In this paper, we present a novel framework for introducing declarative knowl- edge to neural network architectures in order to guide training and prediction. Our frame- work systematically compiles logical state- ments into computation graphs that augment a neural network without extra learnable pa- rameters or manual redesign. We evaluate our modeling strategy on three tasks: ma- chine comprehension, natural language infer- ence, and text chunking. Our experiments show that knowledge-augmented networks can strongly improve over baselines, especially in low-data regimes. 1 Introduction Neural models demonstrate remarkable predic- tive performance across a broad spectrum of NLP tasks: e.g., natural language inference (Parikh et al., 2016), machine comprehension (Seo et al., 2017), machine translation (Bahdanau et al., 2015), and summarization (Rush et al., 2015). These successes can be attributed to their ability to learn robust representations from data. However, such end-to-end training demands a large num- ber of training examples; for example, training a typical network for machine translation may re- quire millions of sentence pairs (e.g. Luong et al., 2015). The difficulties and expense of curating large amounts of annotated data are well under- stood and, consequently, massive datasets may not be available for new tasks, domains or languages. In this paper, we argue that we can combat the data hungriness of neural networks by tak- ing advantage of domain knowledge expressed as Gaius Julius Caesar (July 100 BC – 15 March 44 BC), Roman general, statesman, Consul and notable author of Latin prose, played a critical role in the events that led to the demise of the Roman Republic and the rise of the Roman Empire through his various military campaigns. Paragraph: Question: Which Roman general is known for writing prose? Figure 1: An example of reading comprehension that illustrates alignments/attention. In this paper, we con- sider the problem of incorporating external knowledge about such alignments into training neural networks. first-order logic. As an example, consider the task of reading comprehension, where the goal is to answer a question based on a paragraph of text (Fig. 1). Attention-driven models such as BiDAF (Seo et al., 2017) learn to align words in the question with words in the text as an interme- diate step towards identifying the answer. While alignments (e.g. author to writing) can be learned from data, we argue that models can reduce their data dependence if they were guided by easily stated rules such as: Prefer aligning phrases that are marked as similar according to an external re- source, e.g., ConceptNet (Liu and Singh, 2004). If such declaratively stated rules can be incorporated into training neural networks, then they can pro- vide the inductive bias that can reduce data depen- dence for …

Content truncated to 500 words