On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis

Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 40–46 Brussels, Belgium, November 1, 2018. c⃝2018 Association for Computational Linguistics 40 On the Role of Text Preprocessing in Neural Network Architectures: An Evaluation Study on Text Categorization and Sentiment Analysis Jose Camacho-Collados School of Computer Science and Informatics Cardiff University camachocolladosj@cardiff.ac.uk Mohammad Taher Pilehvar School of Computer Engineering Iran University of Science and Technology pilehvar@iust.ac.ir Abstract Text preprocessing is often the first step in the pipeline of a Natural Language Process- ing (NLP) system, with potential impact in its final performance. Despite its importance, text preprocessing has not received much at- tention in the deep learning literature. In this paper we investigate the impact of simple text preprocessing decisions (particularly tokeniz- ing, lemmatizing, lowercasing and multiword grouping) on the performance of a standard neural text classifier. We perform an extensive evaluation on standard benchmarks from text categorization and sentiment analysis. While our experiments show that a simple tokeniza- tion of input text is generally adequate, they also highlight significant degrees of variabil- ity across preprocessing techniques. This re- veals the importance of paying attention to this usually-overlooked step in the pipeline, partic- ularly when comparing different models. Fi- nally, our evaluation provides insights into the best preprocessing practices for training word embeddings. 1 Introduction Words are often considered as the basic con- stituents of texts for many languages, including English.1 The first module in an NLP pipeline is a tokenizer which transforms texts to sequences of words. However, in practise, other preprocessing techniques can be (and are) further used together with tokenization. These include lemmatization, lowercasing and multiword grouping, among oth- ers. Although these preprocessing decisions have 1Note that although word-based models are mainstream in NLP in general and text classification in particular, recent work has also considered other linguistic units, such as char- acters (Kim et al., 2016; Xiao and Cho, 2016) or word senses (Li and Jurafsky, 2015; Flekova and Gurevych, 2016; Pile- hvar et al., 2017). These techniques require a different kind of preprocessing and, while they have been shown effective in various settings, in this work we only focus on the main- stream word-based models. been studied in the context of conventional text classification techniques (Leopold and Kinder- mann, 2002; Uysal and Gunal, 2014), little at- tention has been paid to them in the more recent neural-based models. The most similar study to ours is Zhang and LeCun (2017), which analyzed different encoding levels for English and Asian languages such as Chinese, Japanese and Korean. As opposed to our work, their analysis was fo- cused on UTF-8 bytes, characters, words, roman- ized characters and romanized words as encoding levels, rather than the preprocessing techniques analyzed in this paper. Additionally, word embeddings have been shown to play an important role in boosting the generalization capabilities of neural systems (Goldberg, 2016; Camacho-Collados and Pile- hvar, 2018). However, while some studies have fo- cused on intrinsically analyzing the role of lemma- tization in …

Content truncated to 500 words