Supervised Morphological Segmentation Using Rich Annotated Lexicon

Proceedings of Recent Advances in Natural Language Processing, pages 52–61, Varna, Bulgaria, Sep 2–4, 2019. https://doi.org/10.26615/978-954-452-056-4_007 52 Supervised Morphological Segmentation Using Rich Annotated Lexicon Ebrahim Ansari,†‡ Zdenˇek ˇZabokrtsk´y,† Mohammad Mahmoudi,‡ Hamid Haghdoost‡ and Jon´aˇs Vidra† † Institute of Formal and Applied Linguistics, Faculty of Mathematics and Physics, Charles University ‡ Department of Computer Science and Information Technology, Institute for Advanced Studies in Basic Sciences {ansari,m.mahmodi,hamid.h}@iasbs.ac.ir {zabokrtsky,vidra}@ufal.mff.cuni.cz Abstract Morphological segmentation of words is the process of dividing a word into smaller units called morphemes; it is tricky es- pecially when a morphologically rich or polysynthetic language is under question. In this work, we designed and evaluated several Recurrent Neural Network (RNN) based models as well as various other ma- chine learning based approaches for the morphological segmentation task. We trained our models using annotated seg- mentation lexicons. To evaluate the effect of the training data size on our models, we decided to create a large hand-annotated morphologically segmented corpus of Per- sian words, which is, to the best of our knowledge, the first and the only seg- mentation lexicon for the Persian lan- guage. In the experimental phase, using the hand-annotated Persian lexicon and two smaller similar lexicons for Czech and Finnish languages, we evaluated the effect of the training data size, different hyper- parameters settings as well as different RNN-based models. 1 Introduction Morphological analysis must be tackled somehow in all natural language processing tasks, such as machine translation, speech recognition, and in- formation retrieval. Morphological segmentation of words is the process of dividing a word into smaller units called morphemes. Morphological segmentation task is harder for those languages which are morphologically rich and complex like Persian, Arabic, Czech, Finnish or Turkish, espe- cially when there are not enough annotated data for those languages. In this paper, we designed and evaluated various supervised setups to per- form morphological segmentation using a hand- annotated segmented lexicon for training. The efficiency of supervised approaches (espe- cially of deep neural network models) is naturally highly dependent on the size of training data. In order to evaluate the effect of the training data size on our segmentation models, we created a rich Persian hand-annotated segmentation lexicon, which is, as far as we know, the first and the only such computer-readable dataset for Persian. Per- sian (Farsi) is one of the languages of the Indo- European language family within the Indo-Iranian branch and is spoken in Iran, Afghanistan, Tajik- istan and some other regions related to ancient Persian. In addition, we evaluated our models on Czech and Finnish, however, the amount of anno- tated data for them is substantially lower. Automatic morphological segmentation was firstly introduced by Harris (1970). More re- cent research on morphological segmentation has been usually focused on unsupervised learning (Goldsmith, 2001; Creutz and Lagus, 2002; Poon et al., 2009; Narasimhan et al., 2015; Cao and Rei, 2016), whose goal is to find the segmenta- tion boundaries using an unlabeled set of word forms (or possibly a corpus too). Probably the most …

Content truncated to 500 words