Rethinking Complex Neural Network Architectures for Document Classification

Proceedings of NAACL-HLT 2019, pages 4046–4051 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 4046 Rethinking Complex Neural Network Architectures for Document Classification Ashutosh Adhikari,∗ Achyudh Ram,∗ Raphael Tang, and Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo {adadhika, arkeshav, r33tang, jimmylin}@uwaterloo.ca Abstract Neural network models for many NLP tasks have grown increasingly complex in recent years, making training and deployment more difficult. A number of recent papers have questioned the necessity of such architectures and found that well-executed, simpler mod- els are quite effective. We show that this is also the case for document classification: in a large-scale reproducibility study of several recent neural models, we find that a simple BiLSTM architecture with appropriate regu- larization yields accuracy and F1 that are ei- ther competitive or exceed the state of the art on four standard benchmark datasets. Sur- prisingly, our simple model is able to achieve these results without attention mechanisms. While these regularization techniques, bor- rowed from language modeling, are not novel, to our knowledge we are the first to apply them in this context. Our work provides an open- source platform and the foundation for future work in document classification. 1 Introduction Recent developments in neural architectures for a wide range of NLP tasks can be character- ized as a drive towards increasingly complex net- work components and modeling techniques. Wor- ryingly, these new models are accompanied by smaller and smaller improvements in effectiveness on standard benchmark datasets, which leads us to wonder if observed improvements are “real”. There is, however, ample evidence to the contrary. To provide a few examples: Melis et al. (2018) report that standard LSTM architectures outper- form more recent models when properly tuned. Vaswani et al. (2017) show that sequence trans- duction using encoder–decoder networks with at- tention mechanisms work just as well with the at- tention module only, making most of the complex ∗ Equal contribution. neural machinery unnecessary. Mohammed et al. (2018) show that simple RNN- and CNN-based models yield accuracies rivaling far more complex architectures in simple question answering over knowledge graphs. Perhaps most damning are the indictments of Sculley et al. (2018), who lament the lack of empirical rigor in our field and cite even more ex- amples where improvements can be attributed to far more mundane reasons (e.g., hyperparameter tuning) or are simply noise. Lipton and Stein- hardt (2018) concur with these sentiments, adding that authors often use fancy mathematics to ob- fuscate or to impress (reviewers) rather than to clarify. Complex architectures are more difficult to train, more sensitive to hyperparameters, and brittle with respect to domains with different data characteristics—thus both exacerbating the “crisis of reproducibility” and making it difficult for prac- titioners to deploy networks that tackle real-world problems in production environments. Like the papers cited above, we question the need for overly complex neural architectures, fo- cusing on the problem of document classification. Starting with a large-scale reproducibility study of several recent neural models, …

Content truncated to 500 words