Learning Distributed Representations of Sentences from Unlabelled Data

Proceedings of NAACL-HLT 2016, pages 1367–1377, San Diego, California, June 12-17, 2016. c⃝2016 Association for Computational Linguistics Learning Distributed Representations of Sentences from Unlabelled Data Felix Hill Computer Laboratory University of Cambridge felix.hill@cl.cam.ac.uk Kyunghyun Cho Courant Institute of Mathematical Sciences & Centre for Data Science New York University kyunghyun.cho@nyu.edu Anna Korhonen Department of Theoretical & Applied Linguistics University of Cambridge alk23@cam.ac.uk Abstract Unsupervised methods for learning distributed representations of words are ubiquitous in to- day’s NLP research, but far less is known about the best ways to learn distributed phrase or sentence representations from unlabelled data. This paper is a systematic comparison of models that learn such representations. We find that the optimal approach depends crit- ically on the intended application. Deeper, more complex models are preferable for rep- resentations to be used in supervised systems, but shallow log-bilinear models work best for building representation spaces that can be decoded with simple spatial distance met- rics. We also propose two new unsupervised representation-learning objectives designed to optimise the trade-off between training time, domain portability and performance. 1 Introduction Distributed representations - dense real-valued vec- tors that encode the semantics of linguistic units - are ubiquitous in today’s NLP research. For single- words or word-like entities, there are established ways to acquire such representations from naturally occurring (unlabelled) training data based on com- paratively task-agnostic objectives (such as predict- ing adjacent words). These methods are well under- stood empirically (Baroni et al., 2014b) and theoret- ically (Levy and Goldberg, 2014). The best word representation spaces reflect consistently-observed aspects of human conceptual organisation (Hill et al., 2015b), and can be added as features to improve the performance of numerous language processing systems (Collobert et al., 2011). By contrast, there is comparatively little consen- sus on the best ways to learn distributed represen- tations of phrases or sentences.1 With the advent of deeper language processing techniques, it is rel- atively common for models to represent phrases or sentences as continuous-valued vectors. Examples include machine translation (Sutskever et al., 2014), image captioning (Mao et al., 2015) and dialogue systems (Serban et al., 2015). While it has been observed informally that the internal sentence rep- resentations of such models can reflect semantic in- tuitions (Cho et al., 2014), it is not known which ar- chitectures or objectives yield the ‘best’ or most use- ful representations. Resolving this question could ultimately have a significant impact on language processing systems. Indeed, it is phrases and sen- tences, rather than individual words, that encode the human-like general world knowledge (or ‘common sense’) (Norman, 1972) that is a critical missing part of most current language understanding systems. We address this issue with a systematic compari- son of cutting-edge methods for learning distributed representations of sentences. We focus on meth- ods that do not require labelled data gathered for the purpose of training models, since such meth- ods are more cost-effective and applicable across languages and domains. We also propose two new phrase or sentence representation learning objec- tives - …

Content truncated to 500 words