Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4375 - 4391 July 10-15, 2022 ©2022 Association for Computational Linguistics Linguistic Frameworks Go Toe-to-Toe at Neuro-Symbolic Language Modeling Jakob Prange Nathan Schneider Georgetown University {jp1724, nathan.schneider}@georgetown.edu Lingpeng Kong The University of Hong Kong lpk@cs.hku.hk Abstract We examine the extent to which, in principle, different syntactic and semantic graph repre- sentations can complement and improve neural language modeling. Specifically, by condition- ing on a subgraph encapsulating the locally rel- evant sentence history, can a model make better next-word predictions than a pretrained sequen- tial language model alone? With an ensem- ble setup consisting of GPT-2 and ground-truth graphs from one of 7 different formalisms, we find that the graph information indeed improves perplexity and other metrics. Moreover, this ar- chitecture provides a new way to compare dif- ferent frameworks of linguistic representation. In our oracle graph setup, training and evalu- ating on English WSJ, semantic constituency structures prove most useful to language mod- eling performance—outpacing syntactic con- stituency structures as well as syntactic and semantic dependency structures. 1 Introduction Linguistic theories posit that humans can take ad- vantage of hierarchical structure related to some notion of compositionality to produce and com- prehend utterances with complex meanings. Yet explicit representations of this kind of structure are harder to come by than raw text, and large-scale pretrained neural language models (e.g., Devlin et al., 2019; Radford et al., 2019) have managed to perform strikingly well at contextually encoding and predicting words from distributional evidence alone. At the same time, there are good reasons to doubt that these models can be said to understand language in any meaningful way (Trott et al., 2020; Bender and Koller, 2020; Merrill et al., 2021). To address this conundrum, people have started to ex- plore probing pretrained models (Liu et al., 2019; Tenney et al., 2019a, inter alia) and supplement- ing training data with linguistic structure guidance (Strubell et al., 2018; Swayamdipta et al., 2018; Peng et al., 2019; Wu et al., 2021, inter alia). A question that has received less attention is which kind of symbolic linguistic representation (SLR) is most conducive to guiding neural lan- guage models (LMs). Numerous domain-general candidates exist (Abend and Rappoport, 2017; Oepen et al., 2019, 2020; Žabokrtský et al., 2020; Müller, 2020): some are focused on syntactic struc- ture, others on semantics (§2; big grey example graphs in the left panels of figure 1). Frameworks vary along several dimensions, with different label inventories and treatments of specific constructions. Formal differences include the type of structure (de- pendency or constituency, one or multiple parents, projectivity) and its relation to the input string. In general, different design choices may aim to cap- ture different kinds of generalizations or facilitate different kinds of processing, and may make pars- ing raw text easier or harder. It is often not obvious which framework should be chosen for best results on an external task—or indeed, how …

Content truncated to 500 words