Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 3799–3812 August 1–6, 2021. ©2021 Association for Computational Linguistics 3799 Assessing the Syntactic Capabilities of Transformer-based Multilingual Language Models Laura P´erez-Mayos1, Alba T´aboas Garc´ıa1, Simon Mille1, Leo Wanner2,1 1 TALN Research Group, Pompeu Fabra University, Barcelona, Spain 2 Catalan Institute for Research and Advanced Studies (ICREA), Barcelona, Spain {laura.perezm|alba.taboas|simon.mille|leo.wanner}@upf.edu Abstract Multilingual Transformer-based language models, usually pretrained on more than 100 languages, have been shown to achieve outstanding results in a wide range of cross- lingual transfer tasks. However, it remains unknown whether the optimization for differ- ent languages conditions the capacity of the models to generalize over syntactic structures, and how languages with syntactic phenomena of different complexity are affected. In this work, we explore the syntactic generalization capabilities of the monolingual and multilin- gual versions of BERT and RoBERTa. More specifically, we evaluate the syntactic gener- alization potential of the models on English and Spanish tests, comparing the syntactic abilities of monolingual and multilingual models on the same language (English), and of multilingual models on two different languages (English and Spanish). For English, we use the available SyntaxGym test suite; for Spanish, we introduce SyntaxGymES, a novel ensemble of targeted syntactic tests in Spanish, designed to evaluate the syntactic generalization capabilities of language models through the SyntaxGym online platform. 1 Introduction Transformer-based neural models such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), DistilBERT (Sanh et al., 2019), XLNet (Yang et al., 2019), etc. are excellent learners. They have been shown to capture a range of different types of lin- guistic information, from morphological (Edmis- ton, 2020) over syntactic (Hewitt and Manning, 2019) to lexico-semantic (Joshi et al., 2020). A particularly significant number of works study the degree to which these models capture and general- ize over (i.e., learn to instantiate correctly in differ- ent contexts) syntactic phenomena, including, e.g., subject-verb agreement, long distance dependen- cies, garden path constructions, etc. (Linzen et al., 2016; Marvin and Linzen, 2018; Futrell et al., 2019; Wilcox et al., 2019a). However, most of these works focus on monolingual models, and, if the coverage of syntactic phenomena is considered sys- tematically and in detail, it is mainly for English, as, e.g., (Hu et al., 2020a). This paper aims to shift the attention from monolingual to multilingual models and to emphasize the importance to also consider the syntactic phenomena of languages other than English when assessing the generalization potential of a model. More specifically, it systematically as- sesses how well multilingual models are capable to generalize over certain syntactic phenomena, com- pared to monolingual models, and how well they can do it not only for English, but also for Spanish. Multilingual models such as mBERT (multilin- gual BERT, (Devlin et al., 2019)), XLM (Lample and Conneau, 2019) and XLM-R (Conneau et al., 2020) proved to achieve outstanding performance on cross-lingual language understanding tasks, in- cluding on low-resource languages for which only little training data is available. However, these models face the risk …

Content truncated to 500 words