Rule-based Automatic Multi-word Term Extraction and Lemmatization

Rule-based Automatic Multi-Word Term Extraction and Lemmatization Ranka Stanković1, Cvetana Krstev2, Ivan Obradović1, Biljana Lazić1, Aleksandra Trtovac3 1University of Belgrade, Faculty of Mining and Geology 2 University of Belgrade, Faculty of Philology 3 University Library “Svetozar Marković”, Belgrade E-mail: ranka.stankovic@rgf.bg.ac.rs, cvetana@poincare.matf.bg.ac.rs, ivan.obradovic@rgf.bg.ac.rs, biljana.lazic@rgf.bg.ac.rs, aleksandra@unilib.bg.ac.rs Abstract In this paper we present a rule-based method for multi-word term extraction that relies on extensive lexical resources in the form of electronic dictionaries and finite-state transducers for modelling various syntactic structures of multi-word terms. The same technology is used for lemmatization of extracted multi-word terms, which is unavoidable for highly inflected languages in order to pass extracted data to evaluators and subsequently to terminological e-dictionaries and databases. The approach is illustrated on a corpus of Serbian texts from the mining domain containing more than 600,000 simple word forms. Extracted and lemmatized multi-word terms are filtered in order to reject falsely offered lemmas and then ranked by introducing measures that combine linguistic and statistical information (C-Value, T-Score, LLR, and Keyness). Mean average precision for retrieval of MWU forms ranges from 0.789 to 0.804, while mean average precision of lemma production ranges from 0.956 to 0.960. The evaluation showed that 94% of distinct multi-word forms were evaluated as proper multi-word units, and among them 97% were associated with correct lemmas. Keywords: term extraction, terminology, multi-word units, lemmatization, finite-state transducers 1. Motivation Various approaches have been proposed for multi-word term (MWT) extraction as this problem has been gaining in importance in the field of Natural Language Processing. Initially, MWT extraction from domain texts has been tackled mainly using the statistical approach based on different statistical measures, following the seminal work of Kenneth Church and Patrick Hanks (1990; 1991) and Frank Smadja (1993). A language independent statistical corpus based term extraction algorithm used on English and Chinese corpora is described in (Pantel&Lin, 2001), while Chen and his associates present a MWT extraction system based on co-related text-segments within a set of documents (Chen et al., 2006). Statistical measures of co-occurrence (MI3 – mutual information) were used for finding MWT candidates in Croatian texts (Tadić&Šojat, 2003). Although the statistical approach has been steadily pursued by a number of researchers, development of lexical resources and local grammars has given impetus to an alternative approach, namely multi-word extraction based on linguistic rules. Recently, a rule-based approach for the extraction of terms was successfully applied on an Arabic scientific and technical corpus, using a cascade of transducers (Ammar et al., 2015). Another example of this approach, SEJFEK, consisting of a grammatical lexicon of about 11,000 Polish MWTs from the economical domain, where inflectional and syntactic variations are described via graph-based rules, is described in (Savary et al., 2012). However, the two approaches are more and more often combined in a hybrid approach. An approach to extracting MWTs from Arabic specialized corpora that uses linguistic rules to parse documents and retrieve candidate terms and statistical measures to deal with ambiguities and rank candidate terms is given in (Bounhas&Slimani, 2009). Several hybrid methods for …

Content truncated to 500 words