Translating Neuralese

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 232–242 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1022 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 232–242 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1022 Translating Neuralese Jacob Andreas Anca Dragan Computer Science DivisionDan Klein University of California, Berkeley {jda,anca,klein}@cs.berkeley.edu Abstract Several approaches have recently been pro- posed for learning decentralized deep mul- tiagent policies that coordinate via a dif- ferentiable communication channel. While these policies are effective for many tasks, interpretation of their induced communi- cation strategies has remained a challenge. Here we propose to interpret agents’ mes- sages by translating them. Unlike in typi- cal machine translation problems, we have no parallel data to learn from. Instead we develop a translation model based on the insight that agent messages and natural lan- guage strings mean the same thing if they induce the same belief about the world in a listener. We present theoretical guarantees and empirical evidence that our approach preserves both the semantics and pragmat- ics of messages by ensuring that players communicating through a translation layer do not suffer a substantial loss in reward rel- ative to players with a common language.1 1 Introduction Several recent papers have described approaches for learning deep communicating policies (DCPs): decentralized representations of behavior that en- able multiple agents to communicate via a differ- entiable channel that can be formulated as a recur- rent neural network. DCPs have been shown to solve a variety of coordination problems, including reference games (Lazaridou et al., 2016b), logic puzzles (Foerster et al., 2016), and simple control (Sukhbaatar et al., 2016). Appealingly, the agents’ communication protocol can be learned via direct 1 We have released code and data at http://github. com/jacobandreas/neuralese. z(1) a z(2) a z(1) b z(2) b Figure 1: Example interaction between a pair of agents in a deep communicating policy. Both cars are attempting to cross the intersection, but cannot see each other. By exchanging message vectors z(t), the agents are able to coordinate and avoid a collision. This paper presents an approach for under- standing the contents of these message vectors by translating them into natural language. backpropagation through the communication chan- nel, avoiding many of the challenging inference problems associated with learning in classical de- centralized decision processes (Roth et al., 2005). But analysis of the strategies induced by DCPs has remained a challenge. As an example, Figure 1 depicts a driving game in which two cars, which are unable to see each other, must both cross an intersection without colliding. In order to ensure success, it is clear that the cars must communi- cate with each other. But a number of successful communication strategies are possible—for exam- ple, they might report their exact (x, y) coordinates at every timestep, or they might simply announce whenever they are entering and leaving the inter- section. If these messages were …

Content truncated to 500 words