Word-order Biases in Deep-agent Emergent Communication

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5166–5175 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5166 Word-order biases in deep-agent emergent communication Rahma Chaabouni1,2, Eugene Kharitonov1, Alessandro Lazaric1, Emmanuel Dupoux1,2 and Marco Baroni1,3 1Facebook A.I. Research 2Cognitive Machine Learning (ENS - EHESS - PSL Research University - CNRS - INRIA) 3ICREA {rchaabouni,kharitonov,lazaric,dpx,mbaroni}@fb.com Abstract Sequence-processing neural networks led to remarkable progress on many NLP tasks. As a consequence, there has been increasing in- terest in understanding to what extent they process language as humans do. We aim here to uncover which biases such models display with respect to “natural” word-order constraints. We train models to communi- cate about paths in a simple gridworld, us- ing miniature languages that reflect or violate various natural language trends, such as the tendency to avoid redundancy or to minimize long-distance dependencies. We study how the controlled characteristics of our miniature languages affect individual learning and their stability across multiple network generations. The results draw a mixed picture. On the one hand, neural networks show a strong tendency to avoid long-distance dependencies. On the other hand, there is no clear preference for the efficient, non-redundant encoding of informa- tion that is widely attested in natural language. We thus suggest inoculating a notion of “ef- fort” into neural networks, as a possible way to make their linguistic behavior more human- like. 1 Introduction Deep neural networks, and in particular “sequence-to-sequence” (Seq2Seq, Sutskever et al., 2014) LSTM recurrent networks, attained astounding successes in many linguistic domains (Goldberg, 2017), but we still have a poor under- standing of their language processing mechanisms (Lake and Baroni, 2018). We study here whether word-order constraints commonly observed in natural language are also found as “inductive” biases in recurrent networks. We consider three such constraints. The first is temporal iconicity, defined as the tendency of clauses denoting events to reflect the chronological order of the denoted events (as in Caesar’s veni, vidi, vici; Greenberg, 1963; Haiman, 1980; Newmeyer, 1992; Radden and Dirven, 2007; Diessel, 2008; Marcus and Calude, 2010; de Ruiter et al., 2018). The second is the need to disambiguate the role of sentence constituents, that can be achieved either by means of fixed-word order (e.g., in an SVO language the first noun phrase denotes the subject), or by overting morphological markers (e.g., the subject is marked with nominative case). As the two mechanisms are redundant, a trade-off is generally observed, where languages preferentially adopt one or the other (Comrie, 1981; Blake, 2001). Finally, we consider the general tendency of languages to avoid or minimize long-distance dependencies (Hawkins, 1994; Gibson, 1998; Futrell et al., 2015). As Futrell et al. (2015) observe, “I checked [it] out”, with one word intervening between the verb and the particle it composes with, ‘is easier or more efficient to produce and comprehend’ than “I checked [the place you recommended] out”, with four intervening words. We test whether such constraints affect LSTM- based Seq2Seq models. To …

Content truncated to 500 words