Natural Language Processing with Small Feed-Forward Networks

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2879–2885 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Natural Language Processing with Small Feed-Forward Networks Jan A. Botha Emily Pitler Ji Ma Anton Bakalov Alex Salcianu David Weiss Ryan McDonald Slav Petrov Google Inc. Mountain View, CA {jabot,epitler,maji,abakalov,salcianu,djweiss,ryanmcd,slav}@google.com Abstract We show that small and shallow feed- forward neural networks can achieve near state-of-the-art results on a range of un- structured and structured language pro- cessing tasks while being considerably cheaper in memory and computational re- quirements than deep recurrent models. Motivated by resource-constrained envi- ronments like mobile phones, we show- case simple techniques for obtaining such small neural network models, and investi- gate different tradeoffs when deciding how to allocate a small memory budget. 1 Introduction Deep and recurrent neural networks with large net- work capacity have become increasingly accurate for challenging language processing tasks. For ex- ample, machine translation models have been able to attain impressive accuracies, with models that use hundreds of millions (Bahdanau et al., 2014; Wu et al., 2016) or billions (Shazeer et al., 2017) of parameters. These models, however, may not be feasible in all computational settings. In partic- ular, models running on mobile devices are often constrained in terms of memory and computation. Long Short-Term Memory (LSTM) mod- els (Hochreiter and Schmidhuber, 1997) have achieved good results with small memory foot- prints by using character-based input representa- tions: e.g., the part-of-speech tagging models of Gillick et al. (2016) have only roughly 900,000 parameters. Latency, however, can still be an is- sue with LSTMs, due to the large number of ma- trix multiplications they require (eight per LSTM cell): Kim and Rush (2016) report speeds of only 8.8 words/second when running a two-layer LSTM translation system on an Android phone. Feed-forward neural networks have the poten- tial to be much faster. In this paper, we show that small feed-forward networks can achieve results at or near the state-of-the-art on a variety of natural language processing tasks, with an order of mag- nitude speedup over an LSTM-based approach. We begin by introducing the network model structure and the character-based representations we use throughout all tasks (§2). The four tasks that we address are: language identification (Lang- ID), part-of-speech (POS) tagging, word segmen- tation, and preordering for translation. In order to use feed-forward networks for structured pre- diction tasks, we use transition systems (Titov and Henderson, 2007, 2010) with feature embeddings as proposed by Chen and Manning (2014), and in- troduce two novel transition systems for the last two tasks. We focus on budgeted models and ab- late four techniques (one on each task) for improv- ing accuracy for a given memory budget: 1. Quantization: Using more dimensions and less precision (Lang-ID: §3.1). 2. Word clusters: Reducing the network size to allow for word clusters and derived features (POS tagging: §3.2). 3. Selected features: Adding explicit feature conjunctions (segmentation: §3.3). 4. Pipelines: Introducing another task in a pipeline and allocating parameters …

Content truncated to 500 words