Unsupervised Lexicon-Based Resolution of Unknown Words for Full Morphological Analysis

Proceedings of ACL-08: HLT, pages 728–736, Columbus, Ohio, USA, June 2008. c⃝2008 Association for Computational Linguistics Unsupervised Lexicon-Based Resolution of Unknown Words for Full Morphological Analysis Meni Adler and Yoav Goldberg and David Gabay and Michael Elhadad Ben Gurion University of the Negev Department of Computer Science∗ POB 653 Be’er Sheva, 84105, Israel {adlerm,goldberg,gabayd,elhadad}@cs.bgu.ac.il Abstract Morphological disambiguation proceeds in 2 stages: (1) an analyzer provides all possible analyses for a given token and (2) a stochastic disambiguation module picks the most likely analysis in context. When the analyzer does not recognize a given token, we hit the prob- lem of unknowns. In large scale corpora, un- knowns appear at a rate of 5 to 10% (depend- ing on the genre and the maturity of the lexi- con). We address the task of computing the distribu- tion p(t|w) for unknown words for full mor- phological disambiguation in Hebrew. We in- troduce a novel algorithm that is language in- dependent: it exploits a maximum entropy let- ters model trained over the known words ob- served in the corpus and the distribution of the unknown words in known tag contexts, through iterative approximation. The algo- rithm achieves 30% error reduction on dis- ambiguation of unknown words over a com- petitive baseline (to a level of 70% accurate full disambiguation of unknown words). We have also verified that taking advantage of a strong language-specific model of morpholog- ical patterns provides the same level of disam- biguation. The algorithm we have developed exploits distributional information latent in a wide-coverage lexicon and large quantities of unlabeled data. ∗This work is supported in part by the Lynn and William Frankel Center for Computer Science. 1 Introduction The term unknowns denotes tokens in a text that can- not be resolved in a given lexicon. For the task of full morphological analysis, the lexicon must pro- vide all possible morphological analyses for any given token. In this case, unknown tokens can be categorized into two classes of missing informa- tion: unknown tokens are not recognized at all by the lexicon, and unknown analyses, where the set of analyses for a lexeme does not contain the cor- rect analysis for a given token. Despite efforts on improving the underlying lexicon, unknowns typi- cally represent 5% to 10% of the number of tokens in large-scale corpora. The alternative to continu- ously investing manual effort in improving the lex- icon is to design methods to learn possible analy- ses for unknowns from observable features: their letter structure and their context. In this paper, we investigate the characteristics of Hebrew unknowns for full morphological analysis, and propose a new method for handling such unavoidable lack of in- formation. Our method generates a distribution of possible analyses for unknowns. In our evaluation, these learned distributions include the correct anal- ysis for unknown words in 85% of the cases, con- tributing an error reduction of over 30% over a com- petitive baseline for the overall task of full morpho- logical analysis in Hebrew. The task …

Content truncated to 500 words