Exploring the Syntactic Abilities of RNNs with Multi-task Learning

Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 3–14, Vancouver, Canada, August 3 - August 4, 2017. c⃝2017 Association for Computational Linguistics Exploring the Syntactic Abilities of RNNs with Multi-task Learning Émile Enguehard1 Yoav Goldberg2 Tal Linzen3,4 1Département d’informatique, ENS, PSL Research University 2Computer Science Department, Bar Ilan University 3LSCP & IJN, CNRS, EHESS and ENS, PSL Research University 4Department of Cognitive Science, Johns Hopkins University {emile.enguehard,tal.linzen}@ens.fr yoav.goldberg@gmail.com Abstract Recent work has explored the syntactic abilities of RNNs using the subject-verb agreement task, which diagnoses sensitiv- ity to sentence structure. RNNs performed this task well in common cases, but fal- tered in complex sentences (Linzen et al., 2016). We test whether these errors are due to inherent limitations of the architec- ture or to the relatively indirect supervi- sion provided by most agreement depen- dencies in a corpus. We trained a sin- gle RNN to perform both the agreement task and an additional task, either CCG su- pertagging or language modeling. Multi- task training led to significantly lower er- ror rates, in particular on complex sen- tences, suggesting that RNNs have the ability to evolve more sophisticated syn- tactic representations than shown before. We also show that easily available agree- ment training data can improve perfor- mance on other syntactic tasks, in partic- ular when only a limited amount of train- ing data is available for those tasks. The multi-task paradigm can also be leveraged to inject grammatical knowledge into lan- guage models. 1 Introduction Recurrent neural networks (RNNs) have seen rapid adoption in natural language processing ap- plications. Since these models are not equipped with explicit linguistic representations such as de- pendency parses or logical forms, new methods are needed to characterize the linguistic general- izations that they capture. One such method is drawn from behavioral psychology: the network is tested on cases that are carefully selected to be informative as to the generalizations that the net- work has acquired. Linzen et al. (2016) have recently applied this methodology to evaluate how well a trained RNN captures sentence structure, using the agreement prediction task (Bock and Miller, 1991; Elman, 1991). The form of an English verb often de- pends on its subject. Identifying the subject of a given verb of requires sensitivity to sentence struc- ture. Consequently, testing an RNN on its ability to choose the correct form of a verb in context can shed light on the sophistication of its syntactic rep- resentations (see Section 2.1 for details). RNNs trained specifically to perform the agree- ment task can achieve very good average per- formance on a corpus, with accuracy close to 99%. However, error rates increase substantially on complex sentences (Linzen et al., 2016, 2017), suggesting that the syntactic knowledge acquired by the RNN is imperfect. Finally, when the RNN is trained as a language model rather than specif- ically on the agreement task, its sensitivity to subject-verb agreement, measured as the relative probability of the grammatical and ungrammatical forms of the verb, …

Content truncated to 500 words