Jump to better conclusions: SCAN both left and right

Jump to better conclusions: SCAN both left and right Jasmijn Bastings1 Marco Baroni2 Jason Weston2 Kyunghyun Cho2,3,4 Douwe Kiela2 1ILLC, University of Amsterdam 2Facebook AI Research 3New York University 4CIFAR Global Scholar bastings@uva.nl {mbaroni,jase,kyunghyuncho,dkiela}@fb.com Abstract Lake and Baroni (2018) recently introduced the SCAN data set, which consists of simple commands paired with action sequences and is intended to test the strong generalization abil- ities of recurrent sequence-to-sequence mod- els. Their initial experiments suggested that such models may fail because they lack the ability to extract systematic rules. Here, we take a closer look at SCAN and show that it does not always capture the kind of generaliza- tion that it was designed for. To mitigate this we propose a complementary dataset, which requires mapping actions back to the original commands, called NACS. We show that mod- els that do well on SCAN do not necessarily do well on NACS, and that NACS exhibits prop- erties more closely aligned with realistic use- cases for sequence-to-sequence models. 1 Introduction In a recent paper, Lake and Baroni (2018) (L&B) investigate if recurrent sequence-to-sequence models can exhibit the same strong generalization that humans are capable of, by virtue of our capacity to infer the meaning of a phrase from its constituent parts (i.e., compositionality), providing empirical tests for this long-standing goal (Fodor and Pylyshyn, 1988). Compositional generalization might be a fundamental component in making models drastically less sample-thirsty than they currently are. L&B introduce the SCAN data set (§2), meant to study such generaliza- tion to novel examples. It consists of simple command-action pairs, in which more complex commands are composed of simpler ones (see Figure 1 for examples). SCAN comprises several tests of generaliza- tion, namely with respect to (1) a random sub- set of the data (‘simple’), (2) commands with ac- tion sequences longer than those seen during train- ing (‘length’), and (3) commands that compose a jump JUMP turn around left LTURN LTURN LTURN LTURN jump thrice and turn left twice JUMP JUMP JUMP LTURN LTURN jump opposite left after walk twice WALK WALK LTURN LTURN JUMP Figure 1: SCAN maps commands to actions primitive in novel ways that was only seen in isola- tion during training (‘primitive’). In the latter case, the training set would for example only include the command ‘jump’, after which the test set includes all other commands containing ‘jump’, e.g. ‘jump opposite left after walk twice’. In this paper we take a closer look at SCAN. We start with the observation (§3) that there are few target-side dependencies in the data, a conse- quence of SCAN being generated from a phrase- structure grammar. We show (§6) that this allows simple sequence-to-sequence models (§5) to ob- tain good accuracies e.g. on tasks involving a new primitive, even without access to previous out- puts. However, these simple models do not use composition in any interesting way, and their per- formance is therefore not a realistic indicator of their generalization capability. We hence propose NACS (§4) as a more …

Content truncated to 500 words