The Effect of Efficient Messaging and Input Variability on Neural-Agent Iterated Language Learning

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10121–10129 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 10121 The Effect of Efficient Messaging and Input Variability on Neural-Agent Iterated Language Learning Yuchen Lian1,2, Arianna Bisazza3, Tessa Verhoef2 1Faculty of Electronic and Information Engineering, Xi’an Jiaotong University 2Leiden Institute of Advanced Computer Science, Leiden University 3Center for Language and Cognition, University of Groningen {y.lian, t.verhoef}@liacs.leidenuniv.nl a.bisazza@rug.nl Abstract Natural languages display a trade-off among different strategies to convey syntactic struc- ture, such as word order or inflection. This trade-off, however, has not appeared in re- cent simulations of iterated language learning with neural network agents (Chaabouni et al., 2019b). We re-evaluate this result in light of three factors that play an important role in comparable experiments from the Language Evolution field: (i) speaker bias towards effi- cient messaging, (ii) non systematic input lan- guages, and (iii) learning bottleneck. Our sim- ulations show that neural agents mainly strive to maintain the utterance type distribution ob- served during learning, instead of developing a more efficient or systematic language. 1 Introduction The world’s languages show immense variety, but linguistic patterns also show important universal tendencies (Greenberg, 1963). It has been argued that these common design features are shaped by human cognitive constraints and pressures dur- ing communication and transmission (Kirby et al., 2014). A well-known example of these tendencies is the trade-off between case marking and word order as redundant strategies to encode the role of sentence constituents (Sinnemäki, 2008; Futrell et al., 2015): flexible order typically correlates with the presence of case marking (e.g. in Russian), while fixed order is often observed in languages with little or no case marking (e.g. English). Researchers interested in language universals and their origins have extensively used agent-based modeling techniques to study the impact of social processes on the emergence of linguistic structures (de Boer, 2006). Besides the horizontal transmis- sion that is often modeled in the referential game setup, the process of iterated learning, where sig- nals are transmitted vertically from generation to generation, has been identified to shape language (Kirby, 2001; Kirby et al., 2014). Recently, the advent of deep learning based NLP has triggered a renewed interest in agent-based sim- ulations of language emergence. Most existing studies simulate the emergence of language by let- ting neural network agents play referential games and studying the signals they use (Kottur et al., 2017; Havrylov and Titov, 2017; Lazaridou et al., 2018; Chaabouni et al., 2019a; Dagan et al., 2021). By contrast, Chaabouni et al. (2019b) expose their agents to a pre-defined language, which is then learned and reproduced iteratively by a chain of agents. They analyze how specific properties of the initial languages affect learnability, and further investigate how agent biases affect the evolution across generations. Among others, they studied whether neural agents tend to avoid redundant cod- ing strategies as natural languages do. However, the case-marking/word-order trade-off did not clearly appear in their iterated learning experiments. In this work, we re-evaluate …

Content truncated to 500 words