The Unstoppable Rise of Computational Linguistics in Deep Learning

Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6294–6306 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 6294 The Unstoppable Rise of Computational Linguistics in Deep Learning James Henderson Idiap Research Institute, Switzerland james.henderson@idiap.ch Abstract In this paper, we trace the history of neural net- works applied to natural language understand- ing tasks, and identify key contributions which the nature of language has made to the devel- opment of neural network architectures. We focus on the importance of variable binding and its instantiation in attention-based models, and argue that Transformer is not a sequence model but an induced-structure model. This perspective leads to predictions of the chal- lenges facing research in deep learning archi- tectures for natural language understanding. 1 Introduction When neural networks first started being applied to natural language in the 1980s and 90s, they repre- sented a radical departure from standard practice in computational linguistics. Connectionists had vector representations and learning algorithms, and they didn’t see any need for anything else. Every- thing was a point in a vector space, and everything about the nature of language could be learned from data. On the other hand, most computational lin- guists had linguistic theories and the poverty-of-the- stimulus argument. Obviously some things were learned from data, but all the interesting things about the nature of language had to be innate. A quarter century later, we can say two things with certainty: they were both wrong. Vector-space representations and machine learning algorithms are much more powerful than was thought. Much of the linguistic knowledge which computational linguists assumed needed to be innate can in fact be learned from data. But the unbounded discrete structured representations they used have not been replaced by vector-space representations. Instead, the successful uses of neural networks in computa- tional linguistics have replaced specific pieces of computational-linguistic models with new neural network architectures which bring together contin- uous vector spaces with structured representations in ways which are novel for both machine learning and computational linguistics. Thus, the great progress which we have made through the application of neural networks to natu- ral language processing should not be viewed as a conquest, but as a compromise. As well as the un- questionable impact of machine learning research on NLP, the nature of language has had a profound impact on progress in machine learning. In this paper we trace this impact, and speculate on future progress and its limits. We start with a sketch of the insights from gram- mar formalisms about the nature of language, with their multiple levels, structured representations and rules. The rules were soon learned with statistical methods, followed by the use of neural networks to replace symbols with induced vectors, but the most effective models still kept structured repre- sentations, such as syntactic trees. More recently, attention-based models have replaced hand-coded structures with induced structures. The resulting models represent language with multiple levels of structured representations, much as has always been done. Given this perspective, …

Content truncated to 500 words