Syntax and Geometry of Information

Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics Volume 1: Long Papers, pages 10576–10595 July 9-14, 2023 ©2023 Association for Computational Linguistics Syntax and Geometry of Information Raphaël Bailly SAMM, EA 4543, FP2M 2036 CNRS Université Paris 1 rbailly@univ-paris1.fr Laurent Leblond Stellantis laurent.leblond1@ stellantis.com Kata Gábor ERTIM, EA 2520 INALCO kata.gabor@inalco.fr Abstract This paper presents an information-theoretical model of syntactic generalization. We study syntactic generalization from the perspective of the capacity to disentangle semantic and struc- tural information, emulating the human capac- ity to assign a grammaticality judgment to se- mantically nonsensical sentences. In order to isolate the structure, we propose to represent the probability distribution behind a corpus as the product of the probability of a semantic con- text and the probability of a structure, the latter being independent of the former. We further elaborate the notion of abstraction as a relax- ation of the property of independence. It is based on the measure of structural and contex- tual information for a given representation. We test abstraction as an optimization objective on the task of inducing syntactic categories from natural language data and show that it signif- icantly outperforms alternative methods. Fur- thermore, we find that when syntax-unaware optimization objectives succeed in the task, their success is mainly due to an implicit dis- entanglement process rather than to the model structure. On the other hand, syntactic cate- gories can be deduced in a principled way from the independence between structure and con- text. 1 Introduction In the context of both human learning and statistical machine learning, what distinguishes generaliza- tion from memorization is the process of deliber- ately ignoring a part of the input information. In machine learning, the generalization will be guided towards some specific direction by the learning hy- pothesis: the model structure and the choice of reg- ularization impacts the nature of the information loss. We can talk about syntactic generalization from textual data when the information pertaining to sentence structure tend to be preserved by the model and the information pertaining to other as- pects of the text tend to be ignored. Syntactic generalization is of great interest be- cause the human capacity to assign an abstract structure to utterances is a prerequisite to creatively combine constituents and understand novel sen- tences (Frege, 1892). Knowledge of syntax can boost the robustness of NLP applications with re- spect to unseen data, in particular when there is a distribution shift (He et al., 2020; Wu et al., 2019). In a broader perspective, understanding syntactic generalization informs the discussion on the learn- ability of syntax from unlabelled text without any built-in grammatical knowledge or inductive bias (Gold, 1967; Clark and Lappin, 2010; Bailly and Gábor, 2020). Finally, studying syntactic gener- alization in large language models (LLMs) sheds light on whether and to what extent these models emulate human functioning with respect to linguis- tic competence. The prevailing formalization of syntax is by means of algebraic compositional rules operating on a finite set of discrete …

Content truncated to 500 words