Unsupervised Tokenization Learning

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3649 - 3664 December 7-11, 2022 ©2022 Association for Computational Linguistics Unsupervised Tokenization Learning Anton Kolonin Aigents SingularityNET Foundation Novosibirsk State University akolonin@gmail.com Vignav Ramesh Harvard University SingularityNET Foundation vignavramesh@college.harvard.edu Abstract In the presented study, we discover that the so- called “transition freedom” metric appears su- perior for unsupervised tokenization purposes in comparison to statistical metrics such as mu- tual information and conditional probability, providing F-measure scores in range from 0.71 to 1.0 across explored multilingual corpora. We find that different languages require different offshoots of that metric (such as derivative, vari- ance, and “peak values”) for successful tok- enization. Larger training corpora do not neces- sarily result in better tokenization quality, while compressing the models by eliminating statis- tically weak evidence tends to improve perfor- mance. The proposed unsupervised tokeniza- tion technique provides quality better than or comparable to lexicon-based ones, depending on the language. 1 Introduction Unsupervised language learning, framed as a prob- lem of language modeling based on unannotated corpora, has attracted great attention in recent years, having achieved significant results with transformer-based models such as BERT and GPT that rely on deep neural networks (DNNs) (Vaswani et al., 2017; Brown et al., 2020). At the same time, the idea of unsupervisedly learning a lan- guage grammar represented “interpretably” via a formal grammar such as Link Grammar has been suggested by Vepstas and Goertzel (2014). Kolonin (2015) proposed yet another approach for the prob- lem: using so-called “deep patterns” with hierar- chical “symbolic” grammatical pattern structures learned from texts as a way to model grammars and ontologies for natural languages suiting a wide range of practical applications. Further studies on this path performed by Glushchenko et al. (2018, 2019) have indicated the possibility of learning grammars as well as domain ontologies given high- quality parse trees of texts obtained from unanno- tated training corpora. Unfortunately, the critical part of the pipelines described in the aforemen- tioned studies was the unsupervised generation of the parses, which turned out to be low-quality by virtue of being based on simple “minimum span- ning trees” either based on mutual information (MI) (see Yuret, 1998) or “contextual information” (see Glushchenko et al., 2019) extracted from a BERT-based deep learning model (Vaswani et al., 2017). Still further studies by Ramesh and Kolonin have demonstrated the possibility of building dif- ferent natural language processing (NLP) applica- tions based on a language model represented by a formal grammar (Link Grammar in the explored cases) (Ramesh and Kolonin, 2020, 2021, 2022). Throughout the course of these studies, the con- cept of “interpretable natural language processing” (INLP) has been introduced to indicate the domain of NLP explorations involving both the learning of language models represented in an interpretable form and the application of these models to dif- ferent tasks such as text segmentation, language generation, and question answering. Yet another problem which has its place in the case of conventional language model learn- ing …

Content truncated to 500 words