Cross-Linguistic Syntactic Evaluation of Word Prediction Models

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5523–5539 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 5523 Cross-Linguistic Syntactic Evaluation of Word Prediction Models Aaron Mueller1 Garrett Nicolai1† Panayiota Petrou-Zeniou2 Natalia Talmina2 Tal Linzen1,2 1Department of Computer Science 2Department of Cognitive Science Johns Hopkins University {amueller, gnicola2, ppetrou1, talmina, tal.linzen}@jhu.edu Abstract A range of studies have concluded that neural word prediction models can distinguish gram- matical from ungrammatical sentences with high accuracy. However, these studies are based primarily on monolingual evidence from English. To investigate how these models’ abil- ity to learn syntax varies by language, we intro- duce CLAMS (Cross-Linguistic Assessment of Models on Syntax), a syntactic evaluation suite for monolingual and multilingual mod- els. CLAMS includes subject-verb agreement challenge sets for English, French, German, Hebrew and Russian, generated from gram- mars we develop. We use CLAMS to evalu- ate LSTM language models as well as mono- lingual and multilingual BERT. Across lan- guages, monolingual LSTMs achieved high ac- curacy on dependencies without attractors, and generally poor accuracy on agreement across object relative clauses. On other constructions, agreement accuracy was generally higher in languages with richer morphology. Multilin- gual models generally underperformed mono- lingual models. Multilingual BERT showed high syntactic accuracy on English, but notice- able deficiencies in other languages. 1 Introduction Neural networks can be trained to predict words from their context with much greater accuracy than the architectures used for this purpose in the past. This has been shown to be the case for both recur- rent neural networks (Mikolov et al., 2010; Sun- dermeyer et al., 2012; Jozefowicz et al., 2016) and non-recurrent attention-based models (Devlin et al., 2019; Radford et al., 2019). To gain a better understanding of these models’ successes and failures, in particular in the domain of syntax, proposals have been made for testing the † Work done while at Johns Hopkins University. Now in the University of British Columbia’s Linguistics Department. models on subsets of the test corpus where success- ful word prediction crucially depends on a correct analysis of the structure of the sentence (Linzen et al., 2016). A paradigmatic example is subject- verb agreement. In many languages, including En- glish, the verb often needs to agree in number (here, singular or plural) with the subject (asterisks repre- sent ungrammatical word predictions): (1) The key to the cabinets is/*are next to the coins. To correctly predict the form of the verb (under- lined), the model needs to determine that the head of the subject of the sentence—an abstract, struc- turally defined notion—is the word key rather than cabinets or coins. The approach of sampling challenging sentences from a test corpus has its limitations. Examples of relevant constructions may be difficult to find in the corpus, and naturally occurring sentences of- ten contain statistical cues (confounds) that make it possible for the model to predict the correct form of the verb without an adequate syntactic analysis (Gu- lordava et al., 2018). To address these …

Content truncated to 500 words