Online Learning of Task-specific Word Representations with a Joint Biconvex Passive-Aggressive Algorithm

Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 775–784, Valencia, Spain, April 3-7, 2017. c⃝2017 Association for Computational Linguistics Online Learning of Task-specific Word Representations with a Joint Biconvex Passive-Aggressive Algorithm Pascal Denis1 and Liva Ralaivola2 1MAGNET, Inria Lille – Nord Europe , Villeneuve d’Ascq, France pascal.denis@inria.fr 2 QARMA, LIF, Aix-Marseille University, Marseille, France liva.ralaivola@lif.univ-mrs.fr Abstract This paper presents a new, efficient method for learning task-specific word vectors using a variant of the Passive- Aggressive algorithm. Specifically, this algorithm learns a word embedding ma- trix in tandem with the classifier param- eters in an online fashion, solving a bi- convex constrained optimization at each it- eration. We provide a theoretical analysis of this new algorithm in terms of regret bounds, and evaluate it on both synthetic data and NLP classification problems, in- cluding text classification and sentiment analysis. In the latter case, we compare various pre-trained word vectors to initial- ize our word embedding matrix, and show that the matrix learned by our algorithm vastly outperforms the initial matrix, with performance results comparable or above the state-of-the-art on these tasks. 1 Introduction Recently, distributed word representations have become a crucial component of many natural language processing systems (Koo et al., 2008; Turian et al., 2010; Collobert et al., 2011). The main appeal of these word embeddings is two- fold: they can be derived directly from raw text data in an unsupervised or weakly-supervised manner, and their latent dimensions condense interesting distributional information about the words, thus allowing for better generalization while also mitigating the presence of rare and un- seen terms. While there are now many different spectral, probabilistic, and deep neural approaches for building vectorial word representations, there is still no clear understanding as to which syn- tactic and semantic information they really cap- ture and whether or how these representations re- ally differ (Chen et al., 2013; Levy and Goldberg, 2014b; Schnabel et al., 2015). Also poorly under- stood is the relation between the word represen- tations and the particular learning algorithm (e.g., whether linear or non-linear) that uses them as in- put (Wang and Manning, 2013). What seems clear, however, is that there is no single best embedding and that their impact is very much task-dependent. This in turn raises the ques- tion of how to learn word representations that are adapted to a particular task and learning objective. Three different research routes have been explored towards learning task-specific word embeddings. A first approach (Collobert et al., 2011; Maas et al., 2011) is to learn the embeddings for the target problem jointly with additional unlabeled or (weakly-)labeled data in a semi-supervised or multi-task approach. While very effective, this joint training typically requires large amounts of data and often prohibitive processing times in the case of multi-layer neural networks (not to men- tion their lack of theoretical learning guarantees in part due to their strong non-convexity). An- other approach consists in training word vectors using some …

Content truncated to 500 words