Learning to Discover, Ground and Use Words with Segmental Neural Language Models

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6429–6441 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6429 Learning to Discover, Ground and Use Words with Segmental Neural Language Models Kazuya Kawakami♠♣ Chris Dyer♣ Phil Blunsom♠♣ ♠Department of Computer Science, University of Oxford, Oxford, UK ♣DeepMind, London, UK {kawakamik,cdyer,pblunsom}@google.com Abstract We propose a segmental neural language model that combines the generalization power of neural networks with the ability to discover word-like units that are latent in unsegmented character sequences. In contrast to previous segmentation models that treat word segmen- tation as an isolated task, our model unifies word discovery, learning how words fit to- gether to form sentences, and, by condition- ing the model on visual context, how words’ meanings ground in representations of non- linguistic modalities. Experiments show that the unconditional model learns predictive dis- tributions better than character LSTM models, discovers words competitively with nonpara- metric Bayesian word segmentation models, and that modeling language conditional on vi- sual context improves performance on both. 1 Introduction How infants discover words that make up their first language is a long-standing question in develop- mental psychology (Saffran et al., 1996). Machine learning has contributed much to this discussion by showing that predictive models of language are capable of inferring the existence of word bound- aries solely based on statistical properties of the input (Elman, 1990; Brent and Cartwright, 1996; Goldwater et al., 2009). However, there are two se- rious limitations of current models of word learning in the context of the broader problem of language acquisition. First, language acquisition involves not only learning what words there are (“the lexicon”), but also how they fit together (“the grammar”). Un- fortunately, the best language models, measured in terms of their ability to predict language (i.e., those which seem acquire grammar best), segment quite poorly (Chung et al., 2017; Wang et al., 2017; Kádár et al., 2018), while the strongest models in terms of word segmentation (Goldwater et al., 2009; Berg-Kirkpatrick et al., 2010) do not ade- quately account for the long-range dependencies that are manifest in language and that are easily captured by recurrent neural networks (Mikolov et al., 2010). Second, word learning involves not only discovering what words exist and how they fit together grammatically, but also determining their non-linguistic referents, that is, their grounding. The work that has looked at modeling acquisition of grounded language from character sequences— usually in the context of linking words to a visu- ally experienced environment—has either explic- itly avoided modeling word units (Gelderloos and Chrupała, 2016) or relied on high-level represen- tations of visual context that overly simplify the richness and ambiguity of the visual signal (John- son et al., 2010; Räsänen and Rasilo, 2015). In this paper, we introduce a single model that discovers words, learns how they fit together (not just locally, but across a complete sentence), and grounds them in learned representations of natu- ralistic non-linguistic visual contexts. We argue that such …

Content truncated to 500 words