Post-Specialisation: Retrofitting Vectors of Words Unseen in Lexical Resources

Proceedings of NAACL-HLT 2018, pages 516–527 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Post-Specialisation: Retrofitting Vectors of Words Unseen in Lexical Resources Ivan Vuli´c1, Goran Glavaš2, Nikola Mrkši´c3, Anna Korhonen1 1 Language Technology Lab, University of Cambridge 2 Data and Web Science Group, University of Mannheim 3 PolyAI {iv250,alk23}@cam.ac.uk goran@informatik.uni-mannheim.de nikola@poly-ai.com Abstract Word vector specialisation (also known as retrofitting) is a portable, light-weight ap- proach to fine-tuning arbitrary distributional word vector spaces by injecting external knowledge from rich lexical resources such as WordNet. By design, these post-processing methods only update the vectors of words oc- curring in external lexicons, leaving the repre- sentations of all unseen words intact. In this paper, we show that constraint-driven vector space specialisation can be extended to unseen words. We propose a novel post-specialisation method that: a) preserves the useful linguistic knowledge for seen words; while b) propagat- ing this external signal to unseen words in or- der to improve their vector representations as well. Our post-specialisation approach explic- its a non-linear specialisation function in the form of a deep neural network by learning to predict specialised vectors from their original distributional counterparts. The learned func- tion is then used to specialise vectors of unseen words. This approach, applicable to any post- processing model, yields considerable gains over the initial specialisation models both in in- trinsic word similarity tasks, and in two down- stream tasks: dialogue state tracking and lexi- cal text simplification. The positive effects per- sist across three languages, demonstrating the importance of specialising the full vocabulary of distributional word vector spaces. 1 Introduction Word representation learning is a key research area in current Natural Language Processing (NLP), with its usefulness demonstrated across a range of tasks (Collobert et al., 2011; Chen and Manning, 2014; Melamud et al., 2016b). The standard tech- niques for inducing distributed word representa- tions are grounded in the distributional hypothesis (Harris, 1954): they rely on co-occurrence informa- tion in large textual corpora (Mikolov et al., 2013b; Pennington et al., 2014; Levy and Goldberg, 2014; Levy et al., 2015; Bojanowski et al., 2017). As a result, these models tend to coalesce the notions of semantic similarity and (broader) conceptual relat- edness, and cannot accurately distinguish antonyms from synonyms (Hill et al., 2015; Schwartz et al., 2015). Recently, we have witnessed a rise of in- terest in representation models that move beyond stand-alone unsupervised learning: they leverage external knowledge in human- and automatically- constructed lexical resources to enrich the semantic content of distributional word vectors, in a process termed semantic specialisation. This is often done as a post-processing (some- times referred to as retrofitting) step: input word vectors are fine-tuned to satisfy linguistic con- straints extracted from lexical resources such as WordNet or BabelNet (Faruqui et al., 2015; Mrkši´c et al., 2017). The use of external curated knowl- edge yields improved word vectors for the benefit of downstream applications (Faruqui, 2016). At the same time, this specialisation of the distribu- tional space distinguishes between …

Content truncated to 500 words