Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 2522–2532 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 2522 Deep Subjecthood: Higher-Order Grammatical Features in Multilingual BERT Isabel Papadimitriou Stanford University isabelvp@stanford.edu Ethan A. Chi Stanford University ethanchi@cs.stanford.edu Richard Futrell University of California, Irvine rfutrell@uci.edu Kyle Mahowald University of California, Santa Barbara mahowald@ucsb.edu Abstract We investigate how Multilingual BERT (mBERT) encodes grammar by examining how the high-order grammatical feature of morphosyntactic alignment (how different languages define what counts as a “subject”) is manifested across the embedding spaces of different languages. To understand if and how morphosyntactic alignment affects contextual embedding spaces, we train classifiers to recover the subjecthood of mBERT embeddings in transitive sentences (which do not contain overt information about morphosyntactic alignment) and then evaluate them zero-shot on intransitive sentences (where subjecthood classification depends on alignment), within and across languages. We find that the resulting classifier distributions reflect the morphosyntactic alignment of their training languages. Our results demonstrate that mBERT representations are influenced by high-level grammatical features that are not manifested in any one input sentence, and that this is robust across languages. Further ex- amining the characteristics that our classifiers rely on, we find that features such as passive voice, animacy and case strongly correlate with classification decisions, suggesting that mBERT does not encode subjecthood purely syntactically, but that subjecthood embedding is continuous and dependent on semantic and discourse factors, as is proposed in much of the functional linguistics literature. To- gether, these results provide insight into how grammatical features manifest in contextual embedding spaces, at a level of abstraction not covered by previous work.1 1 Introduction Our goal is to understand whether, and how, large pretrained models encode abstract features of the 1We release the code to reproduce our experiments here https://github.com/toizzy/deep-subjecthood Figure 1: Top: Illustration of the difference between alignment systems. A (for agent) is notation used for the transitive subject, and O for the transitive ob- ject: “The lawyer chased the dog.” S denotes the intransitive subject: “The lawyer laughed.” The blue circle indicates which roles are marked as “subject” in each system. Bottom: Illustration of the training and test process. We train a classifier to distinguish A from O arguments using the BERT contextual embeddings, and test the classifier’s behavior on intransitive subjects (S). The re- sulting distribution reveals to what extent morphosyn- tactic alignment (above) affects model behavior. grammars of languages. To do so, we analyze the notion of subjecthood in Multilingual BERT (mBERT) across diverse languages with different morphosyntactic alignments. Alignment (how each language defines what classifies as a “sub- ject”) is a feature of the grammar of a language, rather than of any single word or sentence, letting us analyze mBERT’s representation of language- specific high-order grammatical properties. Recent work has demonstrated that transformer models of language, such as BERT (Devlin et al., 2019), encode sentences in structurally meaning- ful ways (Manning et al., 2020; Rogers et al., 2020; Kovaleva et al., …

Content truncated to 500 words