Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5604–5620 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 5604 Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models Yiwen Wang1 Jennifer Hu2 Roger Levy2 Peng Qian2 1 John A. Paulson School of Engineering and Applied Sciences, Harvard University 2 Department of Brain and Cognitive Sciences, MIT yiwenwang@g.harvard.edu {jennhu,rplevy,pqian}@mit.edu Abstract Prior work has shown that structural supervi- sion helps English language models learn gen- eralizations about syntactic phenomena such as subject-verb agreement. However, it re- mains unclear if such an inductive bias would also improve language models’ ability to learn grammatical dependencies in typologically dif- ferent languages. Here we investigate this question in Mandarin Chinese, which has a lo- gographic, largely syllable-based writing sys- tem; different word order; and sparser mor- phology than English. We train LSTMs, Recurrent Neural Network Grammars, Trans- former language models, and Transformer- parameterized generative parsing models on two Mandarin Chinese datasets of different sizes. We evaluate the models’ ability to learn different aspects of Mandarin grammar that as- sess syntactic and semantic relationships. We find suggestive evidence that structural super- vision helps with representing syntactic state across intervening content and improves per- formance in low-data settings, suggesting that the benefits of hierarchical inductive biases in acquiring dependency relationships may ex- tend beyond English. 1 Introduction A rich collection of targeted linguistic evaluations has shown that neural language models can surpris- ingly learn many aspects of grammar from unla- beled linguistic input (e.g., Linzen et al., 2016; Gu- lordava et al., 2018; Warstadt et al., 2020; Hu et al., 2020; Xiang et al., 2021). There is also growing ev- idence that explicit modeling of syntax helps neural network-based language models represent syntactic state and exhibit human-like processing behaviors of non-local grammatical dependencies, including number agreement (Kuncoro et al., 2018), negative polarity licensing, filler-gap dependencies (Wilcox et al., 2019; Hu et al., 2020), and garden-path ef- fects (Futrell et al., 2019; Hu et al., 2020). However, this line of research has focused primarily on the syntax of English. It is unclear to what extent struc- tural supervision may help neural language models generalize for languages with differing typologies. Expanding these analyses beyond English has the potential to inform scientific questions about in- ductive biases for language acquisition, as well as practical questions about model architectures that approach language-independence (Bender, 2011). Here, we perform a controlled case study of grammatical knowledge in Mandarin Chinese lan- guage models. The orthography and grammar of Chinese provide a useful testing ground given the differences from English and other Indo-European languages (see, e.g., Shopen 1985; Li et al. 2015). Whereas today’s Indo-European languages like En- glish generally use phone-based orthography, Chi- nese uses a logographic system where each charac- ter generally responds to a syllable. Most Mandarin Chinese words are one or two syllables, influencing the distribution of tokens. Grammatically, Chinese has almost no inflectional morphology, and corpus studies suggest that the average dependency length of Mandarin Chinese …

Content truncated to 500 words