Revisiting Simple Neural Probabilistic Language Models

Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5181–5188 June 6–11, 2021. ©2021 Association for Computational Linguistics 5181 Revisiting Simple Neural Probabilistic Language Models Simeng Sun and Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst {simengsun, miyyer}@cs.umass.edu Abstract Recent progress in language modeling has been driven not only by advances in neural ar- chitectures, but also through hardware and op- timization improvements. In this paper, we re- visit the neural probabilistic language model (NPLM) of Bengio et al. (2003), which sim- ply concatenates word embeddings within a fixed window and passes the result through a feed-forward network to predict the next word. When scaled up to modern hardware, this model (despite its many limitations) performs much better than expected on word-level lan- guage model benchmarks. Our analysis re- veals that the NPLM achieves lower perplex- ity than a baseline Transformer with short in- put contexts but struggles to handle long-term dependencies. Inspired by this result, we mod- ify the Transformer by replacing its first self- attention layer with the NPLM’s local concate- nation layer, which results in small but con- sistent perplexity decreases across three word- level language modeling datasets. 1 Introduction Over the past decade, state-of-the-art neural ar- chitectures for language modeling (LM) have transitioned from simple recurrent neural net- works (Mikolov et al., 2011) to LSTMs (Zaremba et al., 2014) and finally to Transformers (Vaswani et al., 2017). This progress is not due solely to LM- specific advances, however, as general-purpose upgrades such as residual connections (He et al., 2016) and layer normalization (Ba et al., 2016) have enabled scaling to huge datasets and model sizes (Kaplan et al., 2020) on powerful GPUs. In this paper, we revisit the neural probabilistic language model (NPLM) of Bengio et al. (2003), the first (and simplest) neural architecture proposed for language modeling, through the lens of modern architecture design, hardware, and optimization. Given an input sequence of tokens, the NPLM first concatenates the previous n token embeddings and N × Feed Forward Add & Norm Linear (Adaptive) Softmax The drought had lasted ten for now million Predict: years concatenate Figure 1: A modernized version of the neural proba- bilistic language model of Bengio et al. (2003), which concatenates token embeddings within a fixed local window and feeds them to a stack of feed-forward lay- ers to predict the next token. Our modified version addi- tionally concatenates representations of the distant con- text, which are computed by applying a weighted aver- age to token representations outside the local window. then passes the result through a feed-forward net- work to predict the next token. Due to its small context window and lack of parameter sharing, the NPLM has been rendered obsolete, discarded in favor of LSTMs and Transformers. To what extent are its limitations mitigated by modern design and optimization choices? To an- swer this question, we design an upgraded NPLM featuring increased depth and window size …

Content truncated to 500 words