Term Validation for Vocabulary Construction and Key Term Extraction

Proceedings of Recent Advances in Natural Language Processing, pages 776–780, Hissar, Bulgaria, 12-14 September 2011. Term Validation for Vocabulary Construction and Key Term Extraction Alexander Ulanov Hewlett-Packard Labs alexander.ulanov@hp.com Andrey Simanovsky Hewlett-Packard Labs andrey.simanovsky@hp.com Abstract We extract new terminology from a text by term validation in a dictionary. Our ap- proach is based on estimating probabili- ties for previously unseen terms, i.e. not present in a dictionary. To do this we apply several probabilistic models previ- ously not used for term recognition and propose a new one. We apply restriction of domain similarity on terms used for prob- ability estimation and vary the parameters of the models. Performance of our ap- proach is demonstrated using Wikipedia titles vocabulary. 1 Introduction Keyphrase extraction or automatic term recogni- tion is an important task in the area of informa- tion retrieval. It is used for annotating text arti- cles, tagging documents, etc. Keyphrases facilitate easier searching, browsing documents, detecting topics, classification, adding contextual advertise- ment, and so on. Current methods of term extraction rely either on statistics of terms inside documents or on exter- nal dictionaries. These approaches work relatively well with large texts and with specialized vocabu- laries. The problem arrives when a text contains a lot of cross-domain terms which are essential and vocabulary does not cover them. One option is to use several vocabularies: a very broad one, like Wikipedia or WordNet, and another one very spe- cific, like Burton’s legal thesaurus. Even in this case two types of terms will not be identified: new terms and term collocations. New terms appear in emerging areas, and established thesauri will not catch them. Term collocation means a specific term used in conjunction with a broad-sense term. Usually it is hard to automatically identify if col- location is a new term or not. This paper addresses the problem of detecting new terms in a text that are missing in the dic- tionary in order to enrich it, or to create a new, domain-specific one. 2 State of the art A comprehensive overview and comparison of au- tomatic term recognition (ATR) methods is pre- sented in (Zhang et al., 2008). The generic approach includes chunking or POS-tagging, stop-word removal, and restrict- ing candidate terms to phrases, usually noun- based (Frantzi and Ananiadou, 1999), (Wermter and Hahn, 2005). These candidates are ranked using word statistics or mappings to external dic- tionaries. Word statistics is used to calculate ter- mhood and unithood. Termhood is a measure of term relevancy to the subject domain. Unithood is a measure of words cohesion in a term. Ter- mhood is usually frequency-based, computed us- ing plain TF or TF-IDF (Medelyan and Witten, 2006). Other approaches to termhood compu- tation use a notion of weirdness (Ahmad et al., 2000), which is based on the term frequency in a different domain compared to the subject domain. It is extended to the notions of domain pertinence in (Sclano et al., 2007). In the work of (Wartena et al., 2010) term distributions are …

Content truncated to 500 words