Determining the Unithood of Word Sequences Using a Probabilistic Approach

Determining the Unithood of Word Sequences using a Probabilistic Approach Wilson Wong, Wei Liu and Mohammed Bennamoun School of Computer Science and Software Engineering University of Western Australia Crawley WA 6009 {wilson,wei,bennamou}@csse.uwa.edu.au Abstract Most research related to unithood were con- ducted as part of a larger effort for the deter- mination of termhood. Consequently, nov- elties are rare in this small sub-field of term extraction. In addition, existing work were mostly empirically motivated and derived. We propose a new probabilistically-derived measure, independent of any influences of termhood, that provides dedicated measures to gather linguistic evidence from parsed text and statistical evidence from Google search engine for the measurement of unit- hood. Our comparative study using 1, 825 test cases against an existing empirically- derived function revealed an improvement in terms of precision, recall and accuracy. 1 Introduction Automatic term recognition, also referred to as term extraction or terminology mining, is the process of extracting lexical units from text and filtering them for the purpose of identifying terms which charac- terise certain domains of interest. This process in- volves the determination of two factors: unithood and termhood. Unithood concerns with whether or not a sequence of words should be combined to form a more stable lexical unit. On the other hand, termhood measures the degree to which these sta- ble lexical units are related to domain-specific con- cepts. Unithood is only relevant to complex terms (i.e. multi-word terms) while termhood (Wong et al., 2007a) deals with both simple terms (i.e. single- word terms) and complex terms. Recent reviews by (Wong et al., 2007b) show that existing research on unithood are mostly carried out as a prerequisite to the determination of termhood. As a result, there is only a small number of existing measures dedi- cated to determining unithood. Besides the lack of dedicated attention in this sub-field of term extrac- tion, the existing measures are usually derived from term or document frequency, and are modified as per need. As such, the significance of the different weights that compose the measures usually assume an empirical viewpoint. Obviously, such methods are at most inspired by, but not derived from formal models (Kageura and Umino, 1996). The three objectives of this paper are (1) to sepa- rate the measurement of unithood from the determi- nation of termhood, (2) to devise a probabilistically- derived measure which requires only one thresh- old for determining the unithood of word se- quences using non-static textual resources, and (3) to demonstrate the superior performance of the new probabilistically-derived measure against existing empirical measures. In regards to the first objective, we will derive our probabilistic measure free from any influence of termhood determination. Follow- ing this, our unithood measure will be an indepen- dent tool that is applicable not only to term extrac- tion, but many other tasks in information extraction and text mining. Concerning the second objective, we will devise our new measure, known as the Odds of Unithood (OU), which are derived using Bayes Theorem and founded on …

Content truncated to 500 words