Determining Gains Acquired from Word Embedding Quantitatively Using Discrete Distribution Clustering

Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1847–1856 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1169 Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 1847–1856 Vancouver, Canada, July 30 - August 4, 2017. c⃝2017 Association for Computational Linguistics https://doi.org/10.18653/v1/P17-1169 Determining Gains Acquired from Word Embedding Quantitatively Using Discrete Distribution Clustering Jianbo Ye?, Yanran Li†, Zhaohui Wu‡, James Z. Wang?, Wenjie Li† and Jia Li? ?The Pennsylvania State University, University Park, Pennsylvania †The Hong Kong Polytechnic University, Hong Kong ‡Microsoft Abstract Word embeddings have become widely- used in document analysis. While a large number of models for mapping words to vector spaces have been developed, it remains undetermined how much net gain can be achieved over traditional approaches based on bag-of-words. In this paper, we propose a new document clustering approach by combining any word embedding with a state-of-the-art algorithm for clustering empirical distri- butions. By using the Wasserstein distance between distributions, the word-to-word semantic relationship is taken into account in a principled way. The new clustering method is easy to use and consistently outperforms other methods on a variety of data sets. More importantly, the method provides an effective framework for determining when and how much word embeddings contribute to document anal- ysis. Experimental results with multiple embedding models are reported. 1 Introduction Word embeddings (a.k.a. word vectors) have been broadly adopted for document analysis (Mikolov et al., 2013a,b). The embeddings can be trained from external large-scale corpus and then easily utilized for different data. To a certain degree, the knowledge mined from the corpus, possibly in very intricate ways, is coded in the vector space, Correspondence should be sent to J. Ye (jxy198@psu.edu) and J. Li (jiali@psu.edu). The work was done when Z. Wu was with Penn State. the samples of which are easy to describe and ready for mathematical modeling. Despite the appeal, researchers will be interested in knowing how much gain an embedding can bring forth over the performance achievable by existing bag-of- words based approaches. Moreover, how can the gain be quantified? Such a preliminary evaluation will be carried out before building a sophisticated pipeline of analysis. Almost every document analysis model used in practice is constructed assuming a cer- tain basic representation—bag-of-words or word embeddings—for the sake of computational tractability. For example, after word embed- ding is done, high-level models in the embedded space, such as entity representations, similarity measures, data manifolds, hierarchical structures, language models, and neural architectures, are designed for various tasks. In order to invent or enhance analysis tools, we want to under- stand precisely the pros and cons of the high- level models and the underlying representations. Because the model and the representation are tightly coupled in an analytical system, it is not easy to pinpoint where the gain or loss found in practice comes from. Should the gain be credited to the mechanism of the model or to the use of word embeddings? …

Content truncated to 500 words