An Unsupervised Method for Weighting Finite-state Morphological Analyzers

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 3842–3850 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 3842 An unsupervised method for weighting finite-state morphological analyzers Amr Keleg, Francis M. Tyers, Nicholas Howell, Tommi A. Pirinen Faculty of Engineering, Department of Linguistics, School of Linguistics, Hamburger Zentrum f¨ur Sprachkorpora Ain Shams University, Indiana University, Higher School of Economics, Hamburg University Cairo, Bloomington, Moscow, Hamburg amr.keleg@eng.asu.edu.eg, ftyers@iu.edu, nlhowell@gmail.com, tommi.antero.pirinen@uni-hamburg.de Abstract Morphological analysis is one of the tasks that have been studied for years. Different techniques have been used to develop models for performing morphological analysis. Models based on finite state transducers have proved to be more suitable for languages with low available resources. In this paper, we have developed a method for weighting a morphological analyzer built using finite state transducers in order to disambiguate its results. The method is based on a word2vec model that is trained in a completely unsupervised way using raw untagged corpora and is able to capture the semantic meaning of the words. Most of the methods used for disambiguating the results of a morphological analyzer relied on having tagged corpora that need to manually built. Additionally, the method developed uses information about the token irrespective of its context unlike most of the other techniques that heavily rely on the word’s context to disambiguate its set of candidate analyses. Keywords: FSTs, FST weighting, constraint grammar, word2vec 1. Introduction Morphological Analysis is the task of mapping an input to- ken to its morphemes. The morphological analysis of a to- ken can change depending on its context (e.g.: The word wound is a verb <v> in the sentence “the device is wound with copper wire” and a singular noun <n><sg> in “a knife wound”). Researchers have been working on build- ing models for languages using different techniques. For low resourced language, it’s not an easy task to find / build a sufficiently large tagged corpus that can be used to train supervised machine learning models. Finite state transducers (FSTs) map a set of input sequences to a set of output sequences. A transducer is defined by an input alphabet A, an output alphabet B , a set of states S, a set of initial states I, a set of final states F and a set of tran- sitions between different states of the transducer T where I ∈ S, F ∈ S, T ∈ S × (A ∪ {ϵ}) × (B ∪ {ϵ}) × S. Each transition has an input token, output token, source state and destination state. Given an input sequence, the output se- quences correspond to the set of valid paths between the ini- tial states and final states passing through a set of interme- diate states. Finite state transducers have been used in mor- phologically analyzing text to convert tokens from the sur- face form to the lexical one (Beesley, 2003). These trans- ducers are non-determinstic if multiple output sequences can be generated for the …

Content truncated to 500 words