Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages

Proceedings of NAACL-HLT 2019, pages 3532–3542 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 3532 Studying the Inductive Biases of RNNs with Synthetic Variations of Natural Languages Shauli Ravfogel1 Yoav Goldberg1,2 Tal Linzen3 1Computer Science Department, Bar Ilan University 2Allen Institute for Artificial Intelligence 3Department of Cognitive Science, Johns Hopkins University {shauli.ravfogel, yoav.goldberg}@gmail.com, tal.linzen@jhu.edu Abstract How do typological properties such as word order and morphological case marking af- fect the ability of neural sequence models to acquire the syntax of a language? Cross- linguistic comparisons of RNNs’ syntactic performance (e.g., on subject-verb agreement prediction) are complicated by the fact that any two languages differ in multiple typo- logical properties, as well as by differences in training corpus. We propose a paradigm that addresses these issues: we create syn- thetic versions of English, which differ from English in one or more typological parame- ters, and generate corpora for those languages based on a parsed English corpus. We re- port a series of experiments in which RNNs were trained to predict agreement features for verbs in each of those synthetic languages. Among other findings, (1) performance was higher in subject-verb-object order (as in En- glish) than in subject-object-verb order (as in Japanese), suggesting that RNNs have a re- cency bias; (2) predicting agreement with both subject and object (polypersonal agreement) improves over predicting each separately, sug- gesting that underlying syntactic knowledge transfers across the two tasks; and (3) overt morphological case makes agreement predic- tion significantly easier, regardless of word or- der. 1 Introduction The strong performance of recurrent neural net- works (RNNs) in applied natural language pro- cessing tasks has motivated an array of studies that have investigated their ability to acquire nat- ural language syntax without syntactic annota- tions; these studies have identified both strengths (Linzen et al., 2016; Giulianelli et al., 2018; Gulordava et al., 2018; Kuncoro et al., 2018; van Schijndel and Linzen, 2018; Wilcox et al., 2018) and limitations (Chowdhury and Zampar- elli, 2018; Marvin and Linzen, 2018; Wilcox et al., 2018). Most of the work so far has focused on En- glish, a language with a specific word order and relatively poor morphology. Do the typological properties of a language affect the ability of RNNs to learn its syntactic regularities? Recent studies suggest that they might. Gulordava et al. (2018) evaluated language models on agreement predic- tion in English, Russian, Italian and Hebrew, and found worse performance on English than the other languages. In the other direction, a study on agreement prediction in Basque showed sub- stantially worse average-case performance than reported for English (Ravfogel et al., 2018). Existing cross-linguistic comparisons are dif- ficult to interpret, however. Models were in- evitably trained on a different corpus for each language. The constructions tested can differ across languages (Gulordava et al., 2018). Per- haps most importantly, any two natural languages differ in a number of typological dimensions, such as morphological richness, word order, or explicit case marking. This paper proposes a controlled experimental …

Content truncated to 500 words