Analogs of Linguistic Structure in Deep Representations

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2893–2897 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Analogs of Linguistic Structure in Deep Representations Jacob Andreas and Dan Klein Computer Science Division University of California, Berkeley jda,klein@cs.berkeley.edu Abstract We investigate the compositional struc- ture of message vectors computed by a deep network trained on a communication game. By comparing truth-conditional representations of encoder-produced mes- sage vectors to human-produced refer- ring expressions, we are able to identify aligned (vector, utterance) pairs with the same meaning. We then search for struc- tured relationships among these aligned pairs to discover simple vector space transformations corresponding to nega- tion, conjunction, and disjunction. Our results suggest that neural representations are capable of spontaneously developing a “syntax” with functional analogues to qualitative properties of natural language.1 1 Introduction The past year has seen a renewal of interest in end- to-end learning of communication strategies be- tween pairs of agents represented with deep net- works (Wagner et al., 2003). Approaches of this kind make it possible to learn decentralized poli- cies from scratch (Foerster et al., 2016; Sukhbaatar et al., 2016), with multiple agents coordinating via learned communication protocol. More gener- ally, any encoder–decoder model (Sutskever et al., 2014) can be viewed as implementing an analo- gous communication protocol, with the input en- coding playing the role of a message in an arti- ficial “language” shared by the encoder and de- coder (Yu et al., 2016). Earlier work has found that under suitable conditions, these protocols acquire simple interpretable lexical (Dircks and Stoness, 1999; Lazaridou et al., 2016) and sequential struc- ture (Mordatch and Abbeel, 2017), even without natural language training data. 1 Code and data are available at http://github. com/jacobandreas/rnn-syn. everything but the blue squares λx.¬(square(x) ∧ blue(x)) ✔ ✔ ✔ Human Speaker Logical Evaluator RNN Encoder MLP Decoder ✔ ✔ ✔ ✔ Truth-conditional=repr. e f �e�W �f�W �e�W ′ �f�W ′ W W ′ W Figure 1: Overview of our task. Given a dataset of referring expression games, example human expressions, and their as- sociated logical forms, we compute explicit denotations both for the original task and in other possible tasks—giving rise to a truth-conditional representation of the natural language. We train a recurrent encoder–decoder model to solve the same tasks directly, and use the decoder to generate comparable truth-conditional representations of neural encodings. One of the distinguishing features of natural language is compositionality: the existence of op- erations like negation and coordination that can be applied to utterances with predictable effects on meaning. RNN models trained for natural lan- guage processing tasks have been found to learn representations that encode some of this composi- tional structure—for example, sentence represen- tations for machine translation encode explicit fea- tures for certain syntactic phenomena (Shi et al., 2016) and represent some semantic relationships translationally (Levy et al., 2014). It is thus nat- ural to ask whether these “language-like” struc- tures also arise spontaneously in models trained directly from an environment …

Content truncated to 500 words