Tensor Product Generation Networks for Deep NLP Modeling

Proceedings of NAACL-HLT 2018, pages 1263–1273 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Tensor Product Generation Networks for Deep NLP Modeling Qiuyuan Huang1, Paul Smolensky1, Xiaodong He2, Li Deng1∗, Dapeng Wu3 1Microsoft Research, WA, USA; 2JD AI Research; 3University of Florida, FL, USA {qihua,psmo}@microsoft.com; xiaodong.he@jd.com; l.deng@ieee.org dpwu@ufl.edu Abstract We present a new approach to the design of deep networks for natural language processing (NLP), based on the general technique of Ten- sor Product Representations (TPRs) for encod- ing and processing symbol structures in dis- tributed neural networks. A network architec- ture — the Tensor Product Generation Net- work (TPGN) — is proposed which is ca- pable in principle of carrying out TPR com- putation, but which uses unconstrained deep learning to design its internal representations. Instantiated in a model for image-caption generation, TPGN outperforms LSTM base- lines when evaluated on the COCO dataset. The TPR-capable structure enables interpreta- tion of internal representations and operations, which prove to contain considerable gram- matical content. Our caption-generation model can be interpreted as generating sequences of grammatical categories and retrieving words by their categories from a plan encoded as a distributed representation. 1 Introduction In this paper we introduce a new architecture for natural language processing (NLP). On what type of principles can a computational architecture be founded? It would seem a sound principle to re- quire that the hypothesis space for learning which an architecture provides include network hypothe- ses that are independently known to be suitable for performing the target task. Our proposed ar- chitecture makes available to deep learning net- work configurations that perform natural language generation by use of Tensor Product Represen- tations (TPRs) (Smolensky and Legendre, 2006). Whether learning will create TPRs is unknown in advance, but what we can say with certainty is that the hypothesis space being searched during learn- ∗LD is currently at Citadel. ing includes TPRs as one appropriate solution to the problem. TPRs are a general method for generating vector-space embeddings of complex symbol structures. Prior work has proved that TPRs en- able powerful symbol processing to be carried out using neural network computation (Smolen- sky, 2012). This includes generating parse trees that conform to a grammar (Cho et al., 2017), al- though incorporating such capabilities into deep learning networks such as those developed here re- mains for future work. The architecture presented here relies on simpler use of TPRs to generate sen- tences; grammars are not explicitly encoded here. We test the proposed architecture by applying it to image-caption generation (on the MS-COCO dataset, (COCO, 2017)). The results improve upon a baseline deploying a state-of-the-art LSTM ar- chitecture (Vinyals et al., 2015), and the TPR foundations of the architecture provide greater in- terpretability. Section 2 of the paper reviews TPR. Section 3 presents the proposed architecture, the Tensor Product Generation Network (TPGN). Section 4 describes the particular model we study for im- age captioning, and Section 5 presents the exper- imental results. Importantly, what the model has …

Content truncated to 500 words