Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop

Proceedings of the 43rd Annual Meeting of the ACL, pages 573–580, Ann Arbor, June 2005. c⃝2005 Association for Computational Linguistics Arabic Tokenization, Part-of-Speech Tagging and Morphological Disambiguation in One Fell Swoop Nizar Habash and Owen Rambow Center for Computational Learning Systems Columbia University New York, NY 10115, USA {habash,rambow}@cs.columbia.edu Abstract We present an approach to using a mor- phological analyzer for tokenizing and morphologically tagging (including part- of-speech tagging) Arabic words in one process. We learn classifiers for individual morphological features, as well as ways of using these classifiers to choose among entries from the output of the analyzer. We obtain accuracy rates on all tasks in the high nineties. 1 Introduction Arabic is a morphologically complex language.1 The morphological analysis of a word consists of determining the values of a large number of (or- thogonal) features, such as basic part-of-speech (i.e., noun, verb, and so on), voice, gender, number, infor- mation about the clitics, and so on.2 For Arabic, this gives us about 333,000 theoretically possible com- pletely specified morphological analyses, i.e., mor- phological tags, of which about 2,200 are actually used in the first 280,000 words of the Penn Arabic Treebank (ATB). In contrast, English morphological tagsets usually have about 50 tags, which cover all morphological variation. As a consequence, morphological disambigua- tion of a word in context, i.e., choosing a complete 1We would like to thank Mona Diab for helpful discussions. The work reported in this paper was supported by NSF Award 0329163. The authors are listed in alphabetical order. 2In this paper, we only discuss inflectional morphology. Thus, the fact that the stem is composed of a root, a pattern, and an infi x vocalism is not relevant except as it affects broken plurals and verb aspect. morphological tag, cannot be done successfully us- ing methods developed for English because of data sparseness. Hajiˇc (2000) demonstrates convincingly that morphological disambiguation can be aided by a morphological analyzer, which, given a word with- out any context, gives us the set of all possible mor- phological tags. The only work on Arabic tagging that uses a corpus for training and evaluation (that we are aware of), (Diab et al., 2004), does not use a morphological analyzer. In this paper, we show that the use of a morphological analyzer outperforms other tagging methods for Arabic; to our knowledge, we present the best-performing wide-coverage to- kenizer on naturally occurring input and the best- performing morphological tagger for Arabic. 2 General Approach Arabic words are often ambiguous in their morpho- logical analysis. This is due to Arabic’s rich system of affixation and clitics and the omission of disam- biguating short vowels and other orthographic di- acritics in standard orthography (“undiacritized or- thography”). On average, a word form in the ATB has about 2 morphological analyses. An example of a word with some of its possible analyses is shown in Figure 1. Analyses 1 and 4 are both nouns. They differ in that the first noun has no affixes, while the second …

Content truncated to 500 words