Discontinuous Constituency and BERT: A Case Study of Dutch

Findings of the Association for Computational Linguistics: ACL 2022, pages 3776 - 3785 May 22-27, 2022 c⃝2022 Association for Computational Linguistics Discontinuous Constituency and BERT: A Case Study of Dutch Konstantinos Kogkalidis∗and Gijs Wijnholds∗ Utrecht Institute of Linguistics OTS, Utrecht University k.kogkalidis,g.j.wijnholds@uu.nl Abstract In this paper, we set out to quantify the syntac- tic capacity of BERT in the evaluation regime of non-context free patterns, as occurring in Dutch. We devise a test suite based on a mildly context-sensitive formalism, from which we derive grammars that capture the linguistic phe- nomena of control verb nesting and verb raising. The grammars, paired with a small lexicon, pro- vide us with a large collection of naturalistic utterances, annotated with verb-subject pair- ings, that serve as the evaluation test bed for an attention-based span selection probe. Our results, backed by extensive analysis, suggest that the models investigated fail in the implicit acquisition of the dependencies examined. 1 Introduction Assessing the ability of large-scale language mod- els to automatically acquire aspects of linguistic theory has become a prominent theme in the litera- ture ever since the inception of BERT (Devlin et al., 2019) and its many variants, largely due to their unanticipated performance. Standard practice in- volves attaching BERT to a shallow neural model of low parametric complexity, and training the latter at detecting various linguistic patterns of interest, revealing in the process the amount to which they are encoded within BERT’s representations. The consensus points to BERT-like models having some capacity for syntactic understanding (Rogers et al., 2020). Their contextualized representations encode structural hierarchies (Lin et al., 2019) that can be projected into parse structures, using linear (He- witt and Manning, 2019) or hyperbolic transforma- tions (Chen et al., 2021), from which one can even obtain an accurate reconstruction of the underlying constituency tree (Vilares et al., 2020). Despite their broadening scope, a latent bias per- sists in the insights provided by the probing liter- ature, due to its focus being, by default, on En- ∗ Equal contribution. glish. English, albeit boasting a rich collection of evaluation resources, is characterized by a simple grammar with relatively few complications over the syntactic and morphological axes. Specifically when it comes to syntax, English lies in close prox- imity to a context-free language, a class character- ized by its low rank in terms of formal complexity and expressive power (Chomsky, 1956). Perhaps more importantly, several commonly used evalua- tion test beds, including the Penn Treebank (Klein and Manning, 2001), are in themselves context- free, muddying the territory between probing for acquired syntactic generalization and arbitrating pattern extraction. As such, claims about the syn- tactic skills of language models should not be as- sumed to freely transfer between languages (and, in some cases, even datasets). In this paper, we seek to evaluate BERT in the face of patterns that go beyond context-freeness. We employ a mildly context-sensitive grammar for- malism to generate complex patterns that do not naturally occur in English. We choose instead to experiment on …

Content truncated to 500 words