Can LSTM Learn to Capture Agreement? The Case of Basque

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 98 Can LSTM Learn to Capture Agreement? The Case of Basque Shauli Ravfogel1 and Francis M. Tyers2,3 and Yoav Goldberg1,4 1 Computer Science Department, Bar Ilan University 2 School of Linguistics, Higher School of Economics 3 Department of Linguistics, Indiana University 4 Allen Institute for Artificial Intelligence {shauli.ravfogel, yoav.goldberg}@gmail.com, ftyers@prompsit.com Abstract Sequential neural networks models are pow- erful tools in a variety of Natural Language Processing (NLP) tasks. The sequential nature of these models raises the questions: to what extent can these models implicitly learn hier- archical structures typical to human language, and what kind of grammatical phenomena can they acquire? We focus on the task of agreement predic- tion in Basque, as a case study for a task that requires implicit understanding of sen- tence structure and the acquisition of a com- plex but consistent morphological system. An- alyzing experimental results from two syntac- tic prediction tasks – verb number prediction and suffix recovery – we find that sequential models perform worse on agreement predic- tion in Basque than one might expect on the basis of a previous agreement prediction work in English. Tentative findings based on diag- nostic classifiers suggest the network makes use of local heuristics as a proxy for the hier- archical structure of the sentence. We propose the Basque agreement prediction task as chal- lenging benchmark for models that attempt to learn regularities in human language. 1 Introduction In recent years, recurrent neural network (RNN) models have emerged as a powerful architec- ture for a variety of NLP tasks (Goldberg, 2017). In particular, gated versions, such as Long Short-Term Networks (LSTMs) (Hochreiter and Schmidhuber, 1997) and Gated Recurrent Units (GRU) (Cho et al., 2014; Chung et al., 2014) achieve state-of-the-art results in tasks such as lan- guage modeling, parsing, and machine translation. RNNs were shown to be able to capture long- term dependencies and statistical regularities in in- put sequences (Karpathy et al., 2015; Linzen et al., 2016; Shi et al., 2016; Jurafsky et al., 2018; Gu- lordava et al., 2018). An adequate evaluation of the ability of RNNs to capture syntactic struc- ture requires a use of established benchmarks. A common approach is the use of an annotated cor- pus to learn an explicit syntax-oriented task, such as parsing or shallow parsing (Dyer et al., 2015; Kiperwasser and Goldberg, 2016; Dozat and Man- ning, 2016) . While such an approach does evalu- ate the ability of the model to learn syntax, it has several drawbacks. First, the annotation process relies on human experts and is thus demanding in term of resources. Second, by its very nature, training a model on such a corpus evaluates it on a human-dictated notion of grammatical structure, and is tightly coupled to a linguistic theory. Lastly, the supervised training process on such a corpus provides the network with explicit grammatical la- bels (e.g. …

Content truncated to 500 words