Towards More Natural Artificial Languages

Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL), pages 85 - 94 December 7-8, 2022 ©2022 Association for Computational Linguistics Towards More Natural Artificial Languages Mark Hopkins Department of Computer Science Williams College mh24@williams.edu Abstract A number of papers have recently argued in favor of using artificially generated languages to investigate the inductive biases of linguistic models, or to develop models for low-resource languages with underrepresented typologies. But the promise of artificial languages comes with a caveat: if these artificial languages are not sufficiently reflective of natural language, then using them as a proxy may lead to inac- curate conclusions. In this paper, we take a step towards increasing the realism of artificial language by introducing a variant of indexed grammars that draw their weights from hierar- chical Pitman-Yor processes. We show that this framework generates languages that emulate the statistics of natural language corpora better than the current approach of directly formulat- ing weighted context-free grammars. 1 Introduction In the World Atlas of Linguistic Structures, Dryer (2013) reports that the plurality of world languages follow a subject-object-verb (SOV) word order. However, relatively few SOV languages (Japanese, Turkish, Persian) have a significant Internet foot- print. Today, the Internet is dominated by subject- verb-object (SVO) languages like English, Spanish, and Chinese. The resulting paucity of non-SVO data makes it difficult to study whether linguistic models have an inductive bias towards particular word orders, or to develop models that perform well on low-resource languages from underrepre- sented linguistic families. In recent work, Wang and Eisner (2016), Ravfogel et al. (2019) and White and Cotterell (2021) argue that artificial languages could be an effective tool for addressing challenges like these, enabling researchers to create large cor- pora that manifest targeted linguistic phenomena. An obvious objection presents itself: what if the models aren’t realistic enough? If not, then con- clusions drawn from artificial languages may not transfer to natural languages. One response to this objection would be to abandon the entire enterprise, and with it the potential advantages of simulated data. An alternative is to follow the tradition of other disciplines who model natural systems (e.g. physics, geology, meteorology) and iterate on these models until they are sufficiently good predictors of observed phenomena. In this spirit, this paper builds upon the frame- work of White and Cotterell (2021), who used weighted context-free grammars to construct ar- tificial languages for studying the inductive biases of neural language models towards particular word orders. Observing that their framework did not account for selectional preference (the linguistic phenomenon that head words and their syntactic dependents are not probabilistically independent), we generalize weighted context-free grammars by introducing the weighted random-access indexed grammar, which facilitates the development of ar- tificial languages that manifest selectional prefer- ence. We also present a methodology for building grammars that emulate statistical relationships ob- served in natural language corpora. Inspired by Teh (2006), we use hierarchical Pitman-Yor processes (Pitman and Yor, 1997) as the token-generating distributions for open-class categories (like noun, verb, …

Content truncated to 500 words