On the Spontaneous Emergence of Discrete and Compositional Signals

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4794–4800 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 4794 On the Spontaneous Emergence of Discrete and Compositional Signals Nur Geffen Lan1 Emmanuel Chemla2 Shane Steinert-Threlkeld3 1 Computational Linguistics Lab, Tel Aviv University 2 EHESS, PSL University, CNRS, Ecole Normale Sup´erieure 3 Department of Linguistics, University of Washington nurlan@mail.tau.ac.il chemla@ens.fr shanest@uw.edu Abstract We propose a general framework to study language emergence through signaling games with neural agents. Using a continuous la- tent space, we are able to (i) train using back- propagation, (ii) show that discrete messages nonetheless naturally emerge. We explore whether categorical perception effects follow and show that the messages are not composi- tional. 1 Introduction In a signaling game, artificial agents learn to com- municate to achieve a common goal: a sender sees some piece of information and produces a mes- sage, which is then sent to a receiver that must take some action (Lewis, 1969; Skyrms, 2010). If the action is coherent with the sender’s initial piece of information, the choice of the message and its interpretation is reinforced. For instance, in a refer- ential game, sender and receiver see a set of objects, and the sender knows which of these the receiver must pick; the sender then sends a message to the receiver, who must interpret it to pick up the right object (Lazaridou et al., 2017, 2018; Havrylov and Titov, 2017; Chaabouni et al., 2019). This setting has been used to study the factors influencing the emergence of various fundamen- tal properties of natural language, such as com- positionality (Kirby et al., 2015; Franke, 2016; Steinert-Threlkeld, 2016; Mordatch and Abbeel, 2018; Lazaridou et al., 2018; Choi et al., 2018). In this paper, we add focus on two other so-called ‘de- sign features’ of natural language (Hockett, 1960): discreteness (i.e. words form clusters in acoustic space), and displacement (i.e. efficient communica- tion can occur about objects and facts beyond the immediate context of the conversation). From an implementation point of view, we fol- low the recent literature which has shown that a sig- naling game is essentially an autoencoder setting, with the encoder playing the role of the sender, and the decoder the role of the receiver (see Fig. 1). In this literature, however, the discreteness of the com- munication protocol is assumed, since the networks then traditionally use a (normally sequential and) discrete latent space (Havrylov and Titov, 2017; Chaabouni et al., 2019; Kharitonov et al., 2019). Our main contribution is a generalization of the current implementation of signaling games as au- toencoders. Our implementation covers a broader variety of signaling games, and it crucially incor- porates the possibility of displacement and makes no a priori assumption of discreteness. Our main result is that under appropriate conditions, discrete- ness emerges spontaneously: if the latent space is thought about as a continuous acoustic space, then trained messages form coherent clusters, just like regular words do. We also show that the …

Content truncated to 500 words