Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features

Proceedings of NAACL-HLT 2018, pages 528–540 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Unsupervised Learning of Sentence Embeddings using Compositional n-Gram Features Matteo Pagliardini* Iprova SA, Switzerland mpagliardini@iprova.com Prakhar Gupta* EPFL, Switzerland prakhar.gupta@epfl.ch Martin Jaggi EPFL, Switzerland martin.jaggi@epfl.ch Abstract The recent tremendous success of unsuper- vised word embeddings in a multitude of ap- plications raises the obvious question if simi- lar methods could be derived to improve em- beddings (i.e. semantic representations) of word sequences as well. We present a sim- ple but efficient unsupervised objective to train distributed representations of sentences. Our method outperforms the state-of-the-art unsu- pervised models on most benchmark tasks, highlighting the robustness of the produced general-purpose sentence embeddings. 1 Introduction Improving unsupervised learning is of key impor- tance for advancing machine learning methods, as to unlock access to almost unlimited amounts of data to be used as training resources. The ma- jority of recent success stories of deep learning does not fall into this category but instead relied on supervised training (in particular in the vision domain). A very notable exception comes from the text and natural language processing domain, in the form of semantic word embeddings trained unsupervised (Mikolov et al., 2013b,a; Penning- ton et al., 2014). Within only a few years from their invention, such word representations – which are based on a simple matrix factorization model as we formalize below – are now routinely trained on very large amounts of raw text data, and have become ubiquitous building blocks of a majority of current state-of-the-art NLP applications. While very useful semantic representations are available for words, it remains challenging to pro- duce and learn such semantic embeddings for longer pieces of text, such as sentences, para- graphs or entire documents. Even more so, it re- * indicates equal contribution mains a key goal to learn such general-purpose representations in an unsupervised way. Currently, two contrary research trends have emerged in text representation learning: On one hand, a strong trend in deep-learning for NLP leads towards increasingly powerful and com- plex models, such as recurrent neural networks (RNNs), LSTMs, attention models and even Neu- ral Turing Machine architectures. While ex- tremely strong in expressiveness, the increased model complexity makes such models much slower to train on larger datasets. On the other end of the spectrum, simpler “shallow” models such as matrix factorizations (or bilinear models) can benefit from training on much larger sets of data, which can be a key advantage, especially in the unsupervised setting. Surprisingly, for constructing sentence embed- dings, naively using averaged word vectors was shown to outperform LSTMs (see Wieting et al. (2016b) for plain averaging, and Arora et al. (2017) for weighted averaging). This example shows potential in exploiting the trade-off be- tween model complexity and ability to process huge amounts of text using scalable algorithms, towards the simpler side. In view of this trade- off, our work here further advances unsupervised learning of sentence embeddings. Our proposed model can be seen …

Content truncated to 500 words