Deep Neural Networks with Massive Learned Knowledge

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1670–1679, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics Deep Neural Networks with Massive Learned Knowledge Zhiting Hu, Zichao Yang, Ruslan Salakhutdinov, Eric P. Xing School of Computer Science Carnegie Mellon University {zhitingh,zichaoy,rsalakhu,epxing}@cs.cmu.edu Abstract Regulating deep neural networks (DNNs) with human structured knowledge has shown to be of great benefit for improved accuracy and in- terpretability. We develop a general frame- work that enables learning knowledge and its confidence jointly with the DNNs, so that the vast amount of fuzzy knowledge can be incor- porated and automatically optimized with lit- tle manual efforts. We apply the framework to sentence sentiment analysis, augmenting a DNN with massive linguistic constraints on discourse and polarity structures. Our model substantially enhances the performance using less training data, and shows improved inter- pretability. The principled framework can also be applied to posterior regularization for regu- lating other statistical models. 1 Introduction Deep neural networks (DNNs) have achieved re- markable success in a large variety of application domains (Krizhevsky et al., 2012; Hinton et al., 2012; Bahdanau et al., 2014). However, the power- ful end-to-end learning comes with limitations, in- cluding the requirement on massive amount of la- beled data, uninterpretability of prediction results, and difficulty of incorporating human intentions and domain knowledge. To alleviate these drawbacks, recent work has fo- cused on training DNNs with extra domain-specific features (Collobert et al., 2011), combining ora- cle similarity constraints (Karaletsos et al., 2016), modeling output correlations (Deng et al., 2014), and others. Recently, Hu et al. (2016) proposed a general distillation framework that transfers knowl- edge expressed as first-order logic (FOL) rules into neural networks, where FOL constraints are inte- grated via posterior regularization (Ganchev et al., 2010). Despite the intuitiveness of FOL rules and the impressive performance in various tasks, the approach, as with the previous posterior constraint methods (Ganchev et al., 2010; Liang et al., 2009; Zhu et al., 2014), has been limited to simple a pri- ori fixed constraints with manually selected weights, lacking the ability of inducing and adapting abstract knowledge from data. This issue is further exacer- bated in the context of regulating DNNs that map raw data directly into the label space, leaving a huge semantic gap in between, and making it unfeasible to express rich human knowledge built on the inter- mediate abstract concepts. In this paper, we introduce a generalized frame- work which enables a learning procedure for knowl- edge representations and their weights jointly with the regulated DNN models. This greatly extends the applicability to massive structures in diverse forms, such as structured models and soft logic rules, fa- cilitating practitioners to incorporate rich domain expertise and fuzzy constraints. Specifically, we propose a mutual distillation method that iteratively transfers information between DNN and structured knowledge, resulting in effective integration of the representation learning capacity of DNN and the generalization power of structured knowledge. Our method does not require additional supervision be- yond raw …

Content truncated to 500 words