Synchronized Mediawiki based analyzer dictionary development

The 3rd International Workshop for Computational Linguistics of Uralic Languages, pages 1–7, St. Petersburg, Russia, 23–24 January 2017. c⃝2017 Association for Computational Linguistics Synchronized Mediawiki based analyzer dictionary development Jack Rueter University of Helsinki Department of Modern Languages jack.rueter@helsinki.fi Mika Hämäläinen University of Helsinki Department of Computer Science mika.hamalainen@helsinki.fi Abstract Open-source analyzer dictionary development is being implemented for Skolt Sami, Ingrian, Moksha-Mordvin, etc. in the Helsinki CSC infrastructure; home of the Finnish Kielipankki ’Language Bank’ and Termipankki ’Term Bank’. The proximity of minority-language corpora in need of annotation and the multiple usage of controlled wikimedia-type dictionaries make CSC an attractive site for synchronized transducer dictionary development. The open-source FST develop- ment of Uralic and other minority languages at Giellatekno-Divvun in Tromsø demonstrates a vast potential for reusage of FST-s, only augmented by open- source work in OmorFi, Apertium and Universal Dependency <http://univer- saldependencies.org/#language-urj>. The initial idea is to allow synchronized editing of Giellatekno XML and CSC Wiki structures via github. In addition to allowing for simple lexc LEMMA:STEM CONTINUATION_LEXICON ”TRANS- LATION” ; line exports, the parallel dictionaries will provide for documentation of derivation, morpho-syntactic information on valency and government, seman- tics and etymology. 1 1 1 Introduction Open-source finite-state transducer development and application as we know it today in the Giellatekno infrastructure1 at Tromsø, Norway dates back to the early 1990s. It begins with the morphological description of different Sami languages, grammatical analysis and syntax. Morphological and morphosyntactic description lays the foun- dation for tool building, such as Divvun2, and working solutions attract soft coding for the application of research and tools to other languages. Open-source compilers from HFST3 in Helsinki, are gradually worked into the infrastructure after 2008. With the growth of the research community and tool building diversity comes the practical- ity of reusable resources, descriptions and testing formalisms. As things improve the number of uses and users also increases. One resource, in particular, is the four-fold combination of lemma, stem, pos/continuation lexicon and gloss, afforded by many of the language projects at Giellatekno in lexc code. Lexc code containing multiple nodes of information can be stored in XML files for xsl transformation and project-specific transducer construction. The need for mul- tiple transducers presents itself when tagging strategies are not shared by the mul- tiple projects of a given language. Although the two-level model may be sufficient for all projects, normative labeling, morphosyntactic information and semantic tag needs will vary for TTS, MT, ICALL, spellcheckers and other morphological analyzer projects. The Giellatekno-Divvun strategy has been to utilize one lexc and twolc in all North Sami projects with filters for selecting the necessary code. For this multiple and iterated use of resources, on the contrary, the solution presents itself in XML-format analyzer dictionaries with multiple xsl transformations and the possibility to keep up with semantic wiki strategies being adopted in open wikimedia projects4. At CSC in Helsinki, Finland, a sanat-server has been put into operation and pro- vides access, initially, to the Kielipankki5 (Finnish Language Bank) wordnet and Lu- …

Content truncated to 500 words