Farasa: A New Fast and Accurate Arabic Word Segmenter

Farasa: A New Fast and Accurate Arabic Word Segmenter Kareem Darwish and Hamdy Mubarak Qatar Computing Research Institute Hamad Bin Khalifa University (HBKU), Doha, Qatar {kdarwish,hmubarak}@qf.org.qa Abstract In this paper, we present Farasa (meaning insight in Arabic), which is a fast and accurate Arabic segmenter. Segmentation involves breaking Arabic words into their constituent clitics. Our approach is based on SVMrank using linear kernels. The features that we utilized account for: likelihood of stems, prefixes, suffixes, and their combination; presence in lexicons containing valid stems and named entities; and underlying stem templates. Farasa outperforms or equalizes state-of-the-art Arabic segmenters, namely QATARA and MADAMIRA. Meanwhile, Farasa is nearly one order of magnitude faster than QATARA and two orders of magnitude faster than MADAMIRA. The segmenter should be able to process one billion words in less than 5 hours. Farasa is written entirely in native Java, with no external dependencies, and is open-source. Keywords: Arabic morphology, word segmentation, stemming 1. Introduction Segmenting Arabic words is one of the most important pro- cessing steps for Arabic and is used as a building block for many Arabic natural language processing (NLP) technolo- gies such as search, part-of-speech tagging, parsing, and machine translation. Word segmentation involves break- ing words into their constituent clitics. For example, the word “wktAbnA”1 (���������� meaning: “and our book”) is composed of three clitics “w+ktAb+nA”, namely the con- junction article “w” as prefix, the stem “ktAb”, and pos- sessive pronoun “nA” as suffix. Due to the importance of the problem, many segmenters have appeared in the past 20 years. They ranged from rule-based morphological analyz- ers (Beesley et al., 1989; Beesley, 1996; Buckwalter, 2002; Khoja, 2001), to light stemmers (Aljlayl and Frieder, 2002; Darwish and Oard, 2007), and statistical word segmenters (Darwish, 2002; Habash et al., 2009; Diab, 2009; Darwish et al., 2014). Statistical word segmenters are considered state-of-the-art with reported segmentation accuracy above 98%. In this paper we introduce a new segmenter, called Farasa (meaning: insight in Arabic), that is at par or better than the state-of-the-art and is very fast. It is an SVMrank-based segmenter that uses a variety of features and lexicons to rank different possible segmentations of a word. The fea- tures that we utilized account for: likelihood of stems, pre- fixes, suffixes, and their combination; presence in lexicons containing valid stems and named entities; and underly- ing stem templates. For Farasa, we make the fundamen- tal simplifying assumption that word-context is not neces- sary to find the correct segmentation of a word. Though a word may have multiple valid segmentations, our re- sults show that this assumption minimally impacts our re- sults. We also report evaluation results on a new test set that we developed to measure segmentation accuracy. We also compare to other state-of-the-art segmenters, namely MADAMIRA (Pasha et al., 2014) and QCRI Advanced Tools For ARAbic (QATARA) (Darwish et al., 2014). 1Buckwalter encoding is used exclusively in the paper. We also wanted to compare to the Stanford Arabic seg- menter (Monroe et al., 2014), but …

Content truncated to 500 words