Towards Universal Segmentations: UniSegments 1.0

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1137–1149 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1137 Towards Universal Segmentations: UniSegments 1.0 Zdeněk Žabokrtský, Niyati Bafna, Jan Bodnár, Lukáš Kyjánek, Emil Svoboda, Magda Ševčíková, Jonáš Vidra Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics Malostranské náměstí 25, Prague, Czech Republic {zabokrtsky, bafna, bodnar, kyjanek, svoboda, sevcikova, vidra}@ufal.mff.cuni.cz Abstract Our work aims at developing a multilingual data resource for morphological segmentation. We present a survey of 17 existing data resources relevant for segmentation in 32 languages, and analyze diversity of how individual linguistic phenomena are captured across them. Inspired by the success of Universal Dependencies, we propose a harmonized scheme for segmentation representation, and convert the data from the studied resources into this common scheme. Harmonized versions of resources available under free licenses are published as a collection called UniSegments 1.0. Keywords: morphology, morpheme, morph, segmentation, multilingual language resources 1. Introduction In natural languages, especially in those with rich mor- phology, a huge number of word forms exist (both potential and corpus attested), with some sub-parts of word forms being clearly “recycled” many times. This motivates the linguistic notion of the morpheme, the smallest meaningful unit of language (Aronoff and Fudeman, 2011). In NLP, various reincarnations of a loosely related notion of “subword” are used too, but defined, rather, by technical means (such as segmen- tation resulting from the Byte-Pair Encoding algorithm used in various contemporary deep learning NLP ap- proaches, see e.g. (Sennrich et al., 2016)).1 For numerous languages, we have annotated datasets with varying sizes, underlying models, and annotation quality. The task of morphological segmentation seems relatively straightforward and less sensitive to local lin- guistic traditions than, e.g., syntactic analysis. How- ever, to the best of our knowledge, there is no rea- sonably standardized and widely accepted approach to morphological segmentation that would be comparable to Universal Dependencies (de Marneffe et al., 2021), a widely multilingual collection of dependency tree- banks, which was the source of inspiration for our work. This paper has two goals. First, we analyze diver- sity of existing segmentation data resources. Second, we present a novel harmonized scheme for represent- ing morphological segmentation and convert data from 17 data resources into the common scheme. All our converters are fully automatic, and we do not insert any new manual annotations. However, we also infer some missing parts of information by heuristic approxima- tions (for instance, when inducing morph boundaries given a sequence of morphemes). The remainder of the paper is structured as follows. 1It is open to conjecture whether the linguistic and the NLP views on word form segmentation will eventually con- verge or not. Section 2 summarizes basic linguistic notions relevant for morphological segmentation. Section 3 presents a survey of data resources included in our study, and Sec- tion 4 compares annotation schemes used in them. In Section 5, the harmonization scheme is proposed, and the resulting multilingual data …

Content truncated to 500 words