Analysing Word Representation from the Input and Output Embeddings in Neural Network Language Models

Proceedings of the 24th Conference on Computational Natural Language Learning, pages 442–454 Online, November 19-20, 2020. c⃝2020 Association for Computational Linguistics https://doi.org/10.18653/v1/P17 442 Analysing Word Representation in the Input and Output Layers of Neural Language Models Steven Derby Paul Miller Barry Devereux Queen’s University Belfast, Belfast, United Kingdom {sderby02, p.miller, b.devereux}@qub.ac.uk Abstract Researchers have recently demonstrated that tying the neural weights between the input look-up table and the output classification layer can improve training and lower perplex- ity on sequence learning tasks such as lan- guage modelling. Such a procedure is possi- ble due to the design of the softmax classifi- cation layer, which previous work has shown to comprise a viable set of semantic represen- tations for the model vocabulary, and these these output embeddings are known to per- form well on word similarity benchmarks. In this paper, we make meaningful comparisons between the input and output embeddings and other SOTA distributional models to gain a bet- ter understanding of the types of information they represent. We also construct a new set of word embeddings using the output embed- dings to create locally-optimal approximations for the intermediate representations from the language model. These locally-optimal em- beddings demonstrate excellent performance across all our evaluations. 1 Introduction Neural Language Modelling has recently gained popularity in NLP. A Neural Network Language Model (NNLM) is tasked with learning a condi- tional probability distribution over the occurrences of words in text (Mikolov et al., 2011). This lan- guage modelling objective requires a neural net- work with sufficient capacity to learn meaning- ful linguistic information such as semantic knowl- edge and syntactic structure. Due to their abil- ity to learn these important linguistic phenomena, NNLMs have been successfully employed as an effective method for generative pretraining (Dai and Le, 2015) and transfer learning to other nat- ural language tasks (Peters et al., 2018a; Howard and Ruder, 2018; Radford et al., 2018). As pre- viously suggested by Bengio et al. (2003), Mnih and Hinton (2007) and Mnih and Teh (2012), the weights of the final fully-connected output layer, or output embeddings, which compute the condi- tional probability distribution over the lexicon, also constitute a legitimate set of embedding vectors representing word meaning, as is the case for the input embeddings. This commonality between the input and output layers of the NNLM has motivated researchers to tie these representations together dur- ing training, improving performance on language modelling tasks (Inan et al., 2016; Press and Wolf, 2017). Furthermore, such a procedure is intuitive, since both the input and output embeddings of the network would appear to be performing a similar task of encoding information about lexical content. As described by Inan et al. (2016), they clearly live in an identical semantic space in language mod- els, unlike other machine learning models were the input and output embeddings have no direct link. On the other hand, it would also be reasonable to assume that the output representations require highly task-specific features (Peters et al., 2018a,b; Devlin et al., …

Content truncated to 500 words