Morfessor EM+Prune: Improved Subword Segmentation with Expectation Maximization and Pruning

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3944–3953 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3944 Morfessor EM+Prune: Improved Subword Segmentation with Expectation Maximization and Pruning Stig-Arne Grönroos1, Sami Virpioja2, Mikko Kurimo1 1Department of Signal Processing and Acoustics, Aalto University, Finland 2Department of Digital Humanities, University of Helsinki, Finland {stig-arne.gronroos,mikko.kurimo}@aalto.fi, sami.virpioja@helsinki.fi Abstract Data-driven segmentation of words into subword units has been used in various natural language processing applications such as automatic speech recognition and statistical machine translation for almost 20 years. Recently it has became more widely adopted, as models based on deep neural networks often benefit from subword units even for morphologically simpler languages. In this paper, we discuss and compare training algorithms for a unigram subword model, based on the Expectation Maximization algorithm and lexicon pruning. Using English, Finnish, North Sami, and Turkish data sets, we show that this approach is able to find better solutions to the optimization problem defined by the Morfessor Baseline model than its original recursive training algorithm. The improved optimization also leads to higher morphological segmentation accuracy when compared to a linguistic gold standard. We publish implementations of the new algorithms in the widely-used Morfessor software package. Keywords: Morphology, Statistical and Machine Learning Methods, Language Modelling, Unsupervised learning, Tools, Less-Resourced/Endangered Languages 1. Introduction Subword segmentation has become a standard prepro- cessing step in many neural approaches to natural language processing (NLP) tasks, e.g Neural Machine Translation (NMT) (Sennrich et al., 2015) and Auto- matic Speech Recognition (ASR) (Smit et al., 2017). Word level modeling suffers from sparse statistics, is- sues with Out-of-Vocabulary (OOV) words, and heavy computational cost due to a large vocabulary. Word level modeling is particularly unsuitable for morpho- logically rich languages, but subwords are commonly used for other languages as well. Subword segmen- tation is best suited for languages with agglutinative morphology. While rule-based morphological segmentation systems can achieve high quality, the large amount of human effort needed makes the approach problematic, partic- ularly for low-resource languages. The systems are lan- guage dependent, necessitating use of multiple tools in multilingual setups. As a fast, cheap and effective al- ternative, data-driven segmentation can be learned in a completely unsupervised manner from raw corpora. Unsupervised morphological segmentation saw much research interest until the early 2010’s; for a sur- vey on the methods, see Hammarström and Borin (2011). Semi-supervised segmentation with already small amounts of annotated training data was found to improve the accuracy significantly when compared to a linguistic segmentation; see Ruokolainen et al. (2016) for a survey. While this line of research has been continued in supervised and more grammatically oriented tasks (Cotterell et al., 2017), the more recent work on unsupervised segmentation is less focused on approximating a linguistically motivated segmentation. Instead, the aim has been to tune subword segmenta- tions for particular applications. For example, the sim- ple substitution dictionary based Byte Pair Encoding segmentation algorithm (Gage, 1994), first proposed for NMT by Sennrich et al. (2015), …

Content truncated to 500 words