Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks

Does Syntax Need to Grow on Trees? Sources of Hierarchical Inductive Bias in Sequence-to-Sequence Networks R. Thomas McCoy Department of Cognitive Science Johns Hopkins University tom.mccoy@jhu.edu Robert Frank Department of Linguistics Yale University robert.frank@yale.edu Tal Linzen Department of Cognitive Science Johns Hopkins University tal.linzen@jhu.edu Abstract Learners that are exposed to the same training data might generalize differently due to dif- fering inductive biases. In neural network models, inductive biases could in theory arise from any aspect of the model architecture. We investigate which architectural factors affect the generalization behavior of neural sequence-to-sequence models trained on two syntactic tasks, English question formation and English tense reinflection. For both tasks, the training set is consistent with a gener- alization based on hierarchical structure and a generalization based on linear order. All ar- chitectural factors that we investigated qual- itatively affected how models generalized, including factors with no clear connection to hierarchical structure. For example, LSTMs and GRUs displayed qualitatively different inductive biases. However, the only factor that consistently contributed a hierarchical bias across tasks was the use of a tree- structured model rather than a model with sequential recurrence, suggesting that human- like syntactic generalization requires architec- tural syntactic structure. 1 Introduction Any finite training set is consistent with multiple generalizations. Therefore, the way that a learner generalizes to unseen examples depends not only on the training data but also on properties of the learner. Suppose a learner is told that a blue triangle is an example of a blick. A learner preferring shape-based generalizations would conclude that blick means ‘‘triangle,’’ while a learner preferring color-based generalizations would conclude that blick means ‘‘blue object’’ (Landau et al., 1988). Factors that guide a learner to choose one generalization over another are called inductive biases. What properties of a learner cause it to have a particular inductive bias? We investigate this question with respect to sequence-to-sequence neural networks (Botvinick and Plaut, 2006; Sutskever et al., 2014). As a test case for studying differences in how models generalize, we use the syntactic task of English question formation, such as transforming (1a) into (1b): (1) a. The zebra does chuckle. b. Does the zebra chuckle? Following Chomsky’s (1980) empirical claims about children’s linguistic input, we constrain our training set to be consistent with two possible rules illustrated in Figure 1: MOVE-MAIN (a rule based on hierarchical syntactic structure) and MOVE-FIRST (a rule based on linear order). We then evaluate each trained model on examples where the rules make different predictions, such as (2): given (2a), MOVE-MAIN would generate (2b) while MOVE-FIRST would generate (2c): (2) a. Your zebras that don’t dance do chuckle. b. Do your zebras that don’t dance chuckle? c. Don’t your zebras that dance do chuckle? Since no such examples appear in the training set, a model’s behavior on them reveals which rule the modelis biased toward.This task allows us to study a particular bias, namely, a bias for hierarchical generalization, which is important for models of 125 Transactions of the Association for …

Content truncated to 500 words