Evaluating Neural Word Representations in Tensor-Based Compositional Settings

Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 708–719, October 25-29, 2014, Doha, Qatar. c⃝2014 Association for Computational Linguistics Evaluating Neural Word Representations in Tensor-Based Compositional Settings Dmitrijs Milajevs1 Dimitri Kartsaklis2 Mehrnoosh Sadrzadeh1 Matthew Purver1 1Queen Mary University of London School of Electronic Engineering and Computer Science Mile End Road, London, UK {d.milajevs,m.sadrzadeh,m.purver}@qmul.ac.uk 2University of Oxford Department of Computer Science Parks Road, Oxford, UK dimitri.kartsaklis@cs.ox.ac.uk Abstract We provide a comparative study be- tween neural word representations and traditional vector spaces based on co- occurrence counts, in a number of com- positional tasks. We use three differ- ent semantic spaces and implement seven tensor-based compositional models, which we then test (together with simpler ad- ditive and multiplicative approaches) in tasks involving verb disambiguation and sentence similarity. To check their scala- bility, we additionally evaluate the spaces using simple compositional methods on larger-scale tasks with less constrained language: paraphrase detection and di- alogue act tagging. In the more con- strained tasks, co-occurrence vectors are competitive, although choice of composi- tional method is important; on the larger- scale tasks, they are outperformed by neu- ral word embeddings, which show robust, stable performance across the tasks. 1 Introduction Neural word embeddings (Bengio et al., 2006; Collobert and Weston, 2008; Mikolov et al., 2013a) have received much attention in the dis- tributional semantics community, and have shown state-of-the-art performance in many natural lan- guage processing tasks. While they have been compared with co-occurrence based models in simple similarity tasks at the word level (Levy et al., 2014; Baroni et al., 2014), we are aware of only one work that attempts a comparison of the two approaches in compositional settings (Blacoe and Lapata, 2012), and this is limited to additive and multiplicative composition, compared against composition via a neural autoencoder. The purpose of this paper is to provide a more complete picture regarding the potential of neu- ral word embeddings in compositional tasks, and meaningfully compare them with the traditional distributional approach based on co-occurrence counts. We are especially interested in investi- gating the performance of neural word vectors in compositional models involving general mathe- matical composition operators, rather than in the more task- or domain-specific deep-learning com- positional settings they have generally been used with so far (for example, by Socher et al. (2012), Kalchbrenner and Blunsom (2013) and many oth- ers). In particular, this is the first large-scale study to date that applies neural word representations in tensor-based compositional distributional models of meaning similar to those formalized by Coecke et al. (2010). We test a range of implementations based on this framework, together with additive and multiplicative approaches (Mitchell and Lap- ata, 2008), in a variety of different tasks. Specif- ically, we use the verb disambiguation task of Grefenstette and Sadrzadeh (2011a) and the tran- sitive sentence similarity task of Kartsaklis and Sadrzadeh (2014) as small-scale focused experi- ments on pre-defined sentence structures. Addi- tionally, we evaluate our vector spaces on para- phrase detection (using the Microsoft Research …

Content truncated to 500 words