Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures

Proceedings of the The 12th Joint Conference on Lexical and Computational Semantics (*SEM 2023), pages 456–468 July 13-14, 2023 ©2023 Association for Computational Linguistics Empirical Sufficiency Lower Bounds for Language Modeling with Locally-Bootstrapped Semantic Structures Jakob Prange and Emmanuele Chersoni Department of Chinese and Bilingual Studies, The Hong Kong Polytechnic University 11 Yuk Choi Road, Hung Hom, Kowloon, Hong Kong (China) first.last@polyu.edu.hk Abstract In this work we build upon negative results from an attempt at language modeling with pre- dicted semantic structure, in order to establish empirical lower bounds on what could have made the attempt successful.1 More specifi- cally, we design a concise binary vector rep- resentation of semantic structure at the lexical level and evaluate in-depth how good an incre- mental tagger needs to be in order to achieve better-than-baseline performance with an end- to-end semantic-bootstrapping language model. We envision such a system as consisting of a (pretrained) sequential-neural component and a hierarchical-symbolic component working to- gether to generate text with low surprisal and high linguistic interpretability. We find that (a) dimensionality of the semantic vector represen- tation can be dramatically reduced without los- ing its main advantages and (b) lower bounds on prediction quality cannot be established via a single score alone, but need to take the distri- butions of signal and noise into account. 1 Introduction It is well-established by now that large pretrained Transformer language models (LMs) can obtain de- tectable knowledge about linguistic structure from raw text distributions (Jawahar et al., 2019; Tenney et al., 2019a, inter alia), thus continuing a long line of research in collecting solid empirical ev- idence for the Distributional Hypothesis (Harris, 1954; Firth, 1957). This is often presented in stark contrast to symbolic linguistic theories and repre- sentations, which put more emphasis on higher- level structural principles. In practice, purely neu- ral models have achieved groundbreaking perfor- mances in a wide range of NLP tasks (Devlin et al., 2019; Brown et al., 2020) in a much more scal- able manner than seems to be possible with sym- bolic ones. Still, theoretical linguistic questions 1Our experimental code is available at https:// github.com/jakpra/SufficiencyLowerBounds. But the test may prove to be more sensitive 1 2 4 17 0 6 3 5 2 4 5 3 17 2 1 4 5 5 PRED EFF ACT coref.gram ACT PREC PAT PAT = = 3 2 ? Figure 1: Example of incremental semantic graph slices obtained from a PTG graph and information flow in a (hypothetical) semantic-bootstrapping LM. In this ex- ample, the dark-red-shaded token ‘to’ is the current LM target; the light-yellow-shaded cells to the left and be- low directly influence the LM decision as in P+22; and the pink arrow marked with ? stands for the intermedi- ate slice prediction step, for which we want to establish sufficiency lower bounds. about the relationship between neural implemen- tation and higher-level symbolic patterns are far from being answered definitively. A common crit- icism of purely distributional models is that they generally lack grounding, because they …

Content truncated to 500 words