SPLIT: Smart Preprocessing (Quasi) Language Independent Tool

SPLIT: Smart Preprocessing (Quasi) Language Independent Tool Mohamed Al-Badrashiny, Arfath Pasha†, Mona Diab, Nizar Habash‡ Owen Rambow†, Wael Salloum†, and Ramy Eskander† Department of Computer Science, The George Washington University {badrashiny,mtdiab}@gwu.edu †Center for Computational Learning Systems, Columbia University †{arfath,rambow,wael,reskander}@ccls.columbia.edu ‡ Computational Approaches to Modeling Language Lab, New York University Abu Dhabi ‡nizar.habash@nyu.edu Abstract Text preprocessing is an important and necessary task for all NLP applications. A simple variation in any preprocessing step may drastically affect the final results. Moreover replicability and comparability, as much as feasible, is one of the goals of our scientific enterprise, thus building systems that can ensure the consistency in our various pipelines would contribute significantly to our goals. The problem has become quite pronounced with the abundance of NLP tools becoming more and more available yet with different levels of specifications. In this paper, we present a dynamic unified preprocessing framework and tool, SPLIT, that is highly configurable based on user requirements which serves as a preprocessing tool for several tools at once. SPLIT aims to standardize the implementations of the most important preprocessing steps by allowing for a unified API that could be exchanged across different researchers to ensure complete transparency in replication. The user is able to select the required preprocessing tasks among a long list of preprocessing steps. The user is also able to specify the order of execution which in turn affects the final preprocessing output. Keywords: Text Preprocessing, NLP, Corpus Linguistics 1. Introduction Text preprocessing and preparation plays an important role in all NLP tasks. Uysal and Gunal (2014) show that choos- ing the appropriate preprocessing tasks significantly im- proves classification accuracy. Therefore in any NLP re- search project, experimenting with different preprocessing schemes is an important component of the research space. But due to the various possibilities for text cleaning and preparation, this task could be a significant source of er- ror not to mention pain. Actually, it could also be a source of poor system performance if a certain preprocessing step has been implemented erroneously. Furthermore, differ- ent implementations of preprocessing steps, however mi- nor, could very well result in variation which could impact research replication. It is worth noting that some preprocessing steps are appli- cable only to some genres/languages. For example in the task of punctuation segmentation in English words such as an attached apostophe should not be blindly applied to ro- manized Arabic used in social media known as Arabizi as some of these punctuation marks are in fact part of the or- thography. Such difficulties motivate the need for a simple standard preprocessing framework that has a unified implementation of the most important preprocessing steps taking into con- sideration the different behaviors of various languages and genres. This enables the researcher to focus more on the research point. Some attempts toward this objective have been introduced. PRETO is a preprocessing tool devel- oped specifically for preprocessing Turkish texts only (Tu- nali and Bilgin, 2012). JPreText is another tool that fo- cuses on stemming, stopword …

Content truncated to 500 words