Using Deep Neural Networks to Learn Syntactic Agreement

LiLT volume 15, issue 2 2017 Using Deep Neural Networks to Learn Syntactic Agreement JEAN-PHILLIPE BERNARDY, University of Gothenburg SHALOM LAPPIN, University of Gothenburg, King’s College London, Queen Mary University of London Abstract We consider the extent to which different deep neural network (DNN) configurations can learn syntactic relations, by taking up Linzen et al.’s (2016) work on subject-verb agreement with LSTM RNNs. We test their methods on a much larger corpus than they used (a ⇠24 million example part of the WaCky corpus, instead of their ⇠1.35 million example corpus, both drawn from Wikipedia). We experiment with several different DNN architectures (LSTM RNNs, GRUs, and CNNs), and alternative parameter settings for these systems (vocabulary size, training to test ratio, number of layers, memory size, drop out rate, and lexical embedding dimension size). We also try out our own unsupervised DNN language model. Our results are broadly compatible with those that Linzen et al. report. However, we discovered some interesting, and in some cases, surprising features of DNNs and language models in their performance of the agreement learning task. In particular, we found that DNNs require large vocabularies to form substantive lexical embeddings in order to learn structural patterns. This finding has interesting consequences for our understanding of the way in which DNNs represent syntactic information. It suggests that DNNs learn syntactic patterns more efficiently through rich lexical embeddings, with semantic as well as syntactic cues, than from training on lexically impoverished strings that highlight structural patterns. 1 2 / LILT VOLUME 15, ISSUE 2 2017 1 Introduction The extent to which machine learning systems in general, and neural networks in particular, can learn regularities that depend on syntactic structure is a topic of ongoing debate.1 Subject-verb number agreement is a paradigmatic case of a syntactic feature that depends on structural properties of a sentence. Linzen et al. (2016) train a Long Short Term Memory Recurrent Neural Network (LSTM RNN) on a subset of a Wikipedia corpus to predict the num- ber of a verb. As they observe, the task increases in difficulty in relation to the length of the sequence of NPs with the wrong number feature that occur between a subject and the verb that it controls. They refer to such intervening NPs as attractors.2 In (1) the attractors for the The students-submit pair are indicated in boldface. (1a) The students submit a final project to complete the course. (b) The students enrolled in the program submit a final project to complete the course. (c) The students enrolled in the program in the Department submit a final project to complete the course. (d) The students enrolled in the program in the Department where my colleague teaches submit a final project to complete the course. Linzen et. al use a dependency parser to identify the controlling subject of each verb in their corpus of examples. This identification is necessary to compute the number of attractors, but it is not used in the training for the number prediction task, because …

Content truncated to 500 words