Lexicon Learning for Few Shot Sequence Modeling

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 4934–4946 August 1–6, 2021. ©2021 Association for Computational Linguistics 4934 Lexicon Learning for Few-Shot Neural Sequence Modeling Ekin Akyürek Jacob Andreas Massachusetts Institute of Technology {akyurek,jda}@mit.edu Abstract Sequence-to-sequence transduction is the core problem in language processing applications as diverse as semantic parsing, machine trans- lation, and instruction following. The neural network models that provide the dominant so- lution to these problems are brittle, especially in low-resource settings: they fail to generalize correctly or systematically from small datasets. Past work has shown that many failures of sys- tematic generalization arise from neural mod- els’ inability to disentangle lexical phenomena from syntactic ones. To address this, we aug- ment neural decoders with a lexical transla- tion mechanism that generalizes existing copy mechanisms to incorporate learned, decontex- tualized, token-level translation rules. We de- scribe how to initialize this mechanism using a variety of lexicon learning algorithms, and show that it improves systematic generaliza- tion on a diverse set of sequence modeling tasks drawn from cognitive science, formal se- mantics, and machine translation.1 1 Introduction Humans exhibit a set of structured and remarkably consistent inductive biases when learning from lan- guage data. For example, in both natural language acquisition and toy language-learning problems like the one depicted in Fig. 1, human learners exhibit a preference for systematic and composi- tional interpretation rules (Guasti 2017, Chapter 4; Lake et al. 2019). These inductive biases in turn support behaviors like one-shot learning of new concepts (Carey and Bartlett, 1978). But in natural language processing, recent work has found that state-of-the-art neural models, while highly effec- tive at in-domain prediction, fail to generalize in human-like ways when faced with rare phenomena 1Our code is released under https://github.com/ ekinakyurek/lexical Train Test dax lug wif zup zup fep zup blicket lug ___? dax blicket zup ___? zup kiki dax ___? wif kiki zup ___? r g b y b b b r r r b g b g r g g b b r y y y lug fep dax fep lug blicket wif wif blicket dax lug kiki wif dax kiki lug lexicon Figure 1: A fragment of the Colors dataset from Lake et al. (2019), a simple sequence-to-sequence translation task. The output vocabulary is only the colored circles r , g , b , y . Humans can reliably fill in the miss- ing test labels on the basis of a small training set, but standard neural models cannot. This paper describes a neural sequence model that obtains improved general- ization via a learned lexicon of token translation rules. and small datasets (Lake and Baroni, 2018), pos- ing a fundamental challenge for NLP tools in the low-data regime. Pause for a moment to fill in the missing labels in Fig. 1. While doing so, which training exam- ples did you pay the most attention to? How many times did you find yourself saying means …

Content truncated to 500 words