Evaluation of Unsupervised Compositional Representations

Proceedings of the 27th International Conference on Computational Linguistics, pages 2666–2677 Santa Fe, New Mexico, USA, August 20-26, 2018. 2666 Evaluation of Unsupervised Compositional Representations Hanan Aldarmaki The George Washington University aldarmaki@gwu.edu Mona Diab The George Washington University mtdiab@gwu.edu Abstract We evaluated various compositional models, from bag-of-words representations to compositional RNN-based models, on several extrinsic supervised and unsupervised evaluation benchmarks. Our results confirm that weighted vector averaging can outperform context-sensitive models in most benchmarks, but structural features encoded in RNN models can also be useful in certain classification tasks. We analyzed some of the evaluation datasets to identify the aspects of mean- ing they measure and the characteristics of the various models that explain their performance variance. 1 Introduction Distributed semantic models for words encode latent features that reflect semantic aspects and corre- lations among words. The goal of compositional semantic models is to induce latent semantic repre- sentations that encode the meaning of phrases, sentences, and paragraphs of variable lengths. Some neural architectures such as convolutional (Kim, 2014) and recursive networks (Socher et al., 2013) han- dle variable-length input by identifying shift-invariant features suitable for the classification problem at hand, which makes it possible to skip composition and work directly with the entire space of individual word embeddings. While such models can achieve excellent performance in supervised classification tasks such as sentiment analysis, we are interested in generic unsupervised fixed-length representations for variable-length text sequences so as to efficiently preserve essential semantic content for later use in various supervised and unsupervised settings. Binary bag-of-words are simple and effective representations that serve as a strong baseline in several classification benchmarks (Wang and Manning, 2012). However, they do not exploit the distributional re- lationships among different words, which limits their applicability and generalization when training data are scarce. Additive compositional functions, such as word vector sum or average, are more effective in semantic similarity tasks even when compared with tensor-based compositional functions (Milajevs et al., 2014) and can outperform more complex and better tuned models based on recurrent neural ar- chitectures on out-of-domain data (Wieting et al., 2015a). Yet, averaging also has several drawbacks: unlike binary representations, the individual word identities are lost, and some words that do not carry semantic significance may end up being more prominently represented than essential words. Further- more, additive compositional models disregard sentence structure and word order, which can lead to loss of semantic nuance. To alleviate the first issue, the weights of various words can be adjusted using word frequency statistics (Riedel et al., 2017) or by inducing context-sensitive weights using recurrent neural networks (Wieting and Gimpel, 2017), both of which have been shown to outperform vector aver- aging. Context-sensitive feed-forward neural models like the paragraph vector (Le and Mikolov, 2014) potentially incorporate word order, yet the training objective may not be sufficient to model deeper struc- ture. Sequence encoder-decoder models, on the other hand, can be trained with various sentence-level objectives, such as neural machine translation (NMT) (Sutskever et al., 2014), predicting surrounding …

Content truncated to 500 words