COGS: A Compositional Generalization Challenge Based on Semantic Interpretation

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 9087–9105, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 9087 COGS: A Compositional Generalization Challenge Based on Semantic Interpretation Najoung Kim Johns Hopkins University n.kim@jhu.edu Tal Linzen New York University linzen@nyu.edu Abstract Natural language is characterized by compo- sitionality: the meaning of a complex expres- sion is constructed from the meanings of its constituent parts. To facilitate the evalua- tion of the compositional abilities of language processing architectures, we introduce COGS, a semantic parsing dataset based on a frag- ment of English. The evaluation portion of COGS contains multiple systematic gaps that can only be addressed by compositional gen- eralization; these include new combinations of familiar syntactic structures, or new com- binations of familiar words and familiar struc- tures. In experiments with Transformers and LSTMs, we found that in-distribution accuracy on the COGS test set was near-perfect (96– 99%), but generalization accuracy was sub- stantially lower (16–35%) and showed high sensitivity to random seed (±6–8%). These findings indicate that contemporary standard NLP models are limited in their compositional generalization capacity, and position COGS as a good way to measure progress. 1 Introduction Humans can produce and understand linguistic ex- pressions that they have not encountered before, by systematically combining atomic building blocks (Montague, 1974). For instance, a speaker that knows the meaning of John loves Mary is neces- sarily able to understand Mary loves John, even if the speaker has not heard or uttered this sentence before (Fodor and Pylyshyn, 1988). The discipline of formal semantics concerns itself with character- izing these building blocks, or “primitives”, and the ways in which they combine to construct the meaning of a complex expression (e.g., Figure 1a). To assess the abilities of computational mod- els of language to generalize compositionally, we propose COGS, a COmpositional Generalization Challenge based on Semantic Interpretation, in (a) (b) Figure 1: (a) The meaning of a sentence (right) is com- positionally built up from the meanings of its parts, in accordance with its structure (left). (b) Interpreting a familiar word in a structure it has not appeared in be- fore. In colors: expressions providing the primitive meanings; in bold: expressions providing evidence that definite NPs may appear in both argument positions of a transitive verb. [[x]] denotes the meaning of x. which a model of language is expected to construct a semantic representation of a given English sen- tence (semantic parsing). The key component of this challenge is that the training and evaluation sets systematically differ in their properties, such that success on the evaluation set requires out-of- distribution generalization. Of the many possible ways that a model could systematically fill such gaps, we expect it to do so in a way that is consis- tent with the compositional principles that guide human linguistic generalization. Figure 1b illus- trates how the meaning of the unseen expression The boy loves the hedgehog could be composition- ally inferred from known parts. In this case, the noun phrase …

Content truncated to 500 words