CNNs found to jump around more skillfully than RNNs: Compositional Generalization in Seq2seq Convolutional Networks

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3919–3923 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 3919 CNNs found to jump around more skillfully than RNNs: Compositional generalization in seq2seq convolutional networks Roberto Dess`ı CIMeC, University of Trento roberto.dessi@studenti.unitn.it Marco Baroni ICREA Facebook AI Research mbaroni@fb.com Abstract Lake and Baroni (2018) introduced the SCAN dataset probing the ability of seq2seq models to capture compositional generalizations, such as inferring the meaning of “jump around” 0- shot from the component words. Recurrent networks (RNNs) were found to completely fail the most challenging generalization cases. We test here a convolutional network (CNN) on these tasks, reporting hugely improved per- formance with respect to RNNs. Despite the big improvement, the CNN has however not induced systematic rules, suggesting that the difference between compositional and non- compositional behaviour is not clear-cut. 1 Introduction Recent deep neural network successes rekindled classic debates on their natural language process- ing abilities (e.g., Kirov and Cotterell, 2018; Mc- Coy et al., 2018; Pater, 2018). Lake and Ba- roni (2018) and Loula et al. (2018) proposed the SCAN challenge to directly assess the ability of sequence-to-sequence networks to perform sys- tematic, compositional generalization of linguistic rules. Their results, and those of Bastings et al. (2018), have shown that modern recurrent net- works (gated RNNs, such as LSTMs and GRUs) generalize well to new sequences that resemble those encountered in training, but achieve very low performance when generalization must be sup- ported by a systematic compositional rule, such as “to X twice you X and X” (e.g., to jump twice, you jump and jump again). Non-recurrent models, such as convolutional neural networks (CNNs, Kalchbrenner et al., 2016; Gehring et al., 2016, 2017) and self- attentive models (Vaswani et al., 2017; Chen et al., 2018) have recently reached comparable or better performance than RNNs on machine translation and other benchmarks. Their linguistic proper- ties are however still generally poorly understood. Tang et al. (2018) have shown that RNNs and self- attentive models are better than CNNs at captur- ing long-distance agreement, while self-attentive networks excel at word sense disambiguation. In an extensive comparison, Bai et al. (2018) showed that CNNs generally outperform RNNs, although the differences were typically not huge. We evalu- ate here an out-of-the-box CNN on the most chal- lenging SCAN tasks, and we uncover the surpris- ing fact that CNNs are dramatically better than RNNs at compositional generalization. As they are more cumbersome to train, we leave testing of self-attentive networks to future work. 2 SCAN SCAN studies compositionality in a simple com- mand execution environment framed as a super- vised sequence-to-sequence task. The neural net- work receives word sequences as input, and has to produce the correspondence action sequence. Examples are given in Table 1. Lake and Ba- roni (2018) originally introduced 4 train/test splits, of which we consider 2.1 In the random split, the training set includes 80% of randomly se- lected distinct SCAN commands, with …

Content truncated to 500 words