Better Arabic Parsing: Baselines, Evaluations, and Analysis

Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 394–402, Beijing, August 2010 Better Arabic Parsing: Baselines, Evaluations, and Analysis Spence Green and Christopher D. Manning Computer Science Department, Stanford University {spenceg,manning}@stanford.edu Abstract In this paper, we offer broad insight into the underperformance of Arabic con- stituency parsing by analyzing the inter- play of linguistic phenomena, annotation choices, and model design. First, we iden- tify sources of syntactic ambiguity under- studied in the existing parsing literature. Second, we show that although the Penn Arabic Treebank is similar to other tree- banks in gross statistical terms, annotation consistency remains problematic. Third, we develop a human interpretable gram- mar that is competitive with a latent vari- able PCFG. Fourth, we show how to build better models for three different parsers. Finally, we show that in application set- tings, the absence of gold segmentation lowers parsing performance by 2–5% F1. 1 Introduction It is well-known that constituency parsing mod- els designed for English often do not generalize easily to other languages and treebanks.1 Expla- nations for this phenomenon have included the relative informativeness of lexicalization (Dubey and Keller, 2003; Arun and Keller, 2005), insensi- tivity to morphology (Cowan and Collins, 2005; Tsarfaty and Sima’an, 2008), and the effect of variable word order (Collins et al., 1999). Cer- tainly these linguistic factors increase the diffi- culty of syntactic disambiguation. Less frequently studied is the interplay among language, annota- tion choices, and parsing model design (Levy and Manning, 2003; K¨ubler, 2005). 1The apparent difficulty of adapting constituency mod- els to non-configurational languages has been one motivation for dependency representations (Hajiˇc and Zem´anek, 2004; Habash and Roth, 2009). To investigate the influence of these factors, we analyze Modern Standard Arabic (henceforth MSA, or simply “Arabic”) because of the unusual opportunity it presents for comparison to English parsing results. The Penn Arabic Treebank (ATB) syntactic guidelines (Maamouri et al., 2004) were purposefully borrowed without major modifica- tion from English (Marcus et al., 1993). Further, Maamouri and Bies (2004) argued that the English guidelines generalize well to other languages. But Arabic contains a variety of linguistic phenom- ena unseen in English. Crucially, the conventional orthographic form of MSA text is unvocalized, a property that results in a deficient graphical rep- resentation. For humans, this characteristic can impede the acquisition of literacy. How do addi- tional ambiguities caused by devocalization affect statistical learning? How should the absence of vowels and syntactic markers influence annotation choices and grammar development? Motivated by these questions, we significantly raise baselines for three existing parsing models through better grammar engineering. Our analysis begins with a description of syn- tactic ambiguity in unvocalized MSA text (§2). Next we show that the ATB is similar to other tree- banks in gross statistical terms, but that annotation consistency remains low relative to English (§3). We then use linguistic and annotation insights to develop a manually annotated grammar for Arabic (§4). To facilitate comparison with previous work, we exhaustively evaluate this grammar and two other …

Content truncated to 500 words