Arabic Data Science Toolkit: An API for Arabic Language Feature Extraction

Arabic Data Science Toolkit: An API for Arabic Language Feature Extraction Paul Rodrigues, Valerie Novak, C. Anton Rytting, Julie Yelle, Jennifer Boutz University of Maryland Center for Advanced Study of Language College Park, MD {prr,crytting,jyelle,jboutz}@umd.edu Abstract We introduce Arabic Data Science Toolkit (ADST), a framework for Arabic language feature extraction, designed for data scientists that may not be familiar with Arabic or natural language processing. The functions in the toolkit allow data scientists to extend their algorithms beyond lexical or statistical methods and leverage Arabic-specific linguistic and stylistic features to enhance their systems and enable them to reach performance levels they might receive on languages with more resources, or languages with which they have more familiarity. Keywords: Arabic, Data Science, Social Media 1 Introduction Data scientists working on language processing prob- lems prefer statistical methods, because they are language-independent. Best performing methods in document classification problems, however, utilize fea- tures that require knowledge of the language text being analyzed. This is especially true with Arabic. Sta- tistical methods, such as bag of words, often are not sufficient for Arabic due to the language’s morpholog- ical complexity (Kulick et al., 2006). (One example of morphological knowledge benefiting statistical parsing may be found in Yuval et al. (2013).) Since data sci- entists can already implement linguistic-universal sta- tistical methods with existing software 1, this toolkit focuses on those features that require linguistic or psy- chological knowledge and the resources to drive them. The Arabic Data Science Toolkit (ADST) is developed primarily in Python, using Flask 2 to run as a web ser- vice server. The result is an API accessible via HTTP, allowing access that is not restricted by programming language. Inputs and outputs to the methods are transmitted in a JSON-like format. While intended to run as a server, individual features are accessible via individual scripts implemented in Python and Perl. Data scientists may throw a large number of features at a classification problem, without initial regards to efficiency, and use feature selection methods to deter- mine which of these features provide the most signal to their problems. They can then choose a subset of those features that are the most useful for a produc- tion system, in order to speed up execution time in a repetitive or realtime/online use case. We designed ADST for the flexibility to extract all features, feature sets, or specific individual features, as required. 1Such as scikit-learn’s CountVectorizer //scikit-learn.org/stable/modules/generated/http: sklearn.feature_extraction.text.CountVectorizer. html 2http://flask.pocoo.org Language-specific resources are somewhat difficult to find for those unfamiliar with a given language or with natural language processing (NLP) in general. Toolk- its such as the Natural Language Toolkit (NLTK)3 make this easier by uniting disparate tools under one framework. Toolkits designed for multiple languages, such as NLTK, tend to be popular, due to both ease of discovery, and language-flexibility. These toolkits may link to tools with language-specific models, but they tend to prefer linking to tools with open-source licensing. Due to licensing restrictions, they may not always integrate the highest performing Arabic …

Content truncated to 500 words