Sparse Coding of Neural Word Embeddings for Multilingual Sequence Labeling

Sparse Coding of Neural Word Embeddings for Multilingual Sequence Labeling G´abor Berend Department of Informatics University of Szeged 2 ´Arp´ad t´er, 6720 Szeged, Hungary berendg@inf.u-szeged.hu Abstract In this paper we propose and carefully eval- uate a sequence labeling framework which solely utilizes sparse indicator features de- rived from dense distributed word represen- tations. The proposed model obtains (near) state-of-the art performance for both part-of- speech tagging and named entity recognition for a variety of languages. Our model relies only on a few thousand sparse coding-derived features, without applying any modification of the word representations employed for the different tasks. The proposed model has fa- vorable generalization properties as it retains over 89.8% of its average POS tagging accu- racy when trained at 1.2% of the total available training data, i.e. 150 sentences per language. 1 Introduction Determining the linguistic structure of natural lan- guage texts based on rich hand-crafted features has a long-going history in natural language processing. The focus of traditional approaches has mostly been on building linguistic analyzers for a particular kind of analysis, which often leads to the incorporation of extensive linguistic and/or domain knowledge for defining the feature space. Consequently, traditional models easily become language and/or task specific resulting in improper generalization properties. A new research direction has emerged recently, that aims at building more general models that re- quire far less feature engineering or none at all. These advancements in natural language processing, pioneered by Bengio et al. (2003), followed by Col- lobert and Weston (2008), Collobert et al. (2011), Mikolov et al. (2013a) among others, employ a dif- ferent philosophy. The objective of these works is to find representations for linguistic phenomena in an unsupervised manner by relying on large amounts of text. Natural language phenomena are extremely sparse by their nature, whereas continuous word em- beddings employ dense representations of words. In our paper we empirically verify via rigorous exper- iments that turning these dense representations into a much sparser (yet denser than one-hot encoding) form can keep the most salient parts of word repre- sentations that are highly suitable for sequence mod- els. Furthermore, our experiments reveal that our pro- posed model performs substantially better than tra- ditional feature-rich models in the absence of abun- dant training data. Our proposed model also has the advantage of performing well on multiple sequence labeling tasks without any modification in the ap- plied word representations thanks to the sparse fea- tures derived from continuous word representations. Our work aims at introducing a novel sequence la- beling model solely utilizing features derived from the sparse coding of continuous word embeddings. Even though sparse coding had previously been uti- lized in NLP prior to us (Faruqui et al., 2015; Chen et al., 2016), to the best of our knowledge, we are the first to propose a sequence labeling framework incorporating it with the following contributions: • We show that the proposed sparse represen- tation is general as sequence labeling models trained on them achieve (near) state-of-the-art performances …

Content truncated to 500 words