Harnessing Deep Neural Networks with Logic Rules

Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 2410–2420, Berlin, Germany, August 7-12, 2016. c⃝2016 Association for Computational Linguistics Harnessing Deep Neural Networks with Logic Rules Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, Eric P. Xing School of Computer Science Carnegie Mellon University {zhitingh,xuezhem,liu,epxing}@cs.cmu.edu, hovy@cmu.edu Abstract Combining deep neural networks with structured logic rules is desirable to harness flexibility and reduce uninterpretability of the neural models. We propose a general framework capable of enhancing various types of neural networks (e.g., CNNs and RNNs) with declarative first-order logic rules. Specifically, we develop an iterative distillation method that transfers the struc- tured information of logic rules into the weights of neural networks. We deploy the framework on a CNN for sentiment anal- ysis, and an RNN for named entity recog- nition. With a few highly intuitive rules, we obtain substantial improvements and achieve state-of-the-art or comparable re- sults to previous best-performing systems. 1 Introduction Deep neural networks provide a powerful mech- anism for learning patterns from massive data, achieving new levels of performance on image classification (Krizhevsky et al., 2012), speech recognition (Hinton et al., 2012), machine trans- lation (Bahdanau et al., 2014), playing strategic board games (Silver et al., 2016), and so forth. Despite the impressive advances, the widely- used DNN methods still have limitations. The high predictive accuracy has heavily relied on large amounts of labeled data; and the purely data-driven learning can lead to uninterpretable and some- times counter-intuitive results (Szegedy et al., 2014; Nguyen et al., 2015). It is also difficult to encode human intention to guide the models to capture de- sired patterns, without expensive direct supervision or ad-hoc initialization. On the other hand, the cognitive process of hu- man beings have indicated that people learn not only from concrete examples (as DNNs do) but also from different forms of general knowledge and rich experiences (Minksy, 1980; Lake et al., 2015). Logic rules provide a flexible declarative language for communicating high-level cognition and expressing structured knowledge. It is there- fore desirable to integrate logic rules into DNNs, to transfer human intention and domain knowledge to neural models, and regulate the learning process. In this paper, we present a framework capable of enhancing general types of neural networks, such as convolutional networks (CNNs) and recurrent networks (RNNs), on various tasks, with logic rule knowledge. Combining symbolic representations with neural methods have been considered in dif- ferent contexts. Neural-symbolic systems (Garcez et al., 2012) construct a network from a given rule set to execute reasoning. To exploit a priori knowl- edge in general neural architectures, recent work augments each raw data instance with useful fea- tures (Collobert et al., 2011), while network train- ing, however, is still limited to instance-label super- vision and suffers from the same issues mentioned above. Besides, a large variety of structural knowl- edge cannot be naturally encoded in the feature- label form. Our framework enables a neural network to learn simultaneously from labeled instances as well as logic …

Content truncated to 500 words