Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 244–253 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Ngram2vec: Learning Improved Word Representations from Ngram Co-occurrence Statistics Zhe Zhao1,2 helloworld@ruc.edu.cn Tao Liu1,2 tliu@ruc.edu.cn Shen Li3,4 shen@mail.bnu.edu.cn Bofang Li1,2 libofang@ruc.edu.cn Xiaoyong Du1,2 duyong@ruc.edu.cn 1 School of Information, Renmin University of China 2 Key Laboratory of Data Engineering and Knowledge Engineering, MOE 3 Institute of Chinese Information Processing, Beijing Normal University 4 UltraPower-BNU Joint Laboratory for Artificial Intelligence, Beijing Normal University Abstract The existing word representation method- s mostly limit their information source to word co-occurrence statistics. In this pa- per, we introduce ngrams into four repre- sentation methods: SGNS, GloVe, PPMI matrix, and its SVD factorization. Com- prehensive experiments are conducted on word analogy and similarity tasks. The results show that improved word repre- sentations are learned from ngram co- occurrence statistics. We also demonstrate that the trained ngram representations are useful in many aspects such as finding antonyms and collocations. Besides, a novel approach of building co-occurrence matrix is proposed to alleviate the hard- ware burdens brought by ngrams. 1 Introduction Recently, deep learning approaches have achieved state-of-the-art results on a range of NLP tasks. One of the most fundamental work in this field is word embedding, where low-dimensional word representations are learned from unlabeled corpo- ra through neural models. The trained word em- beddings reflect semantic and syntactic informa- tion of words. They are not only useful in reveal- ing lexical semantics, but also used as inputs of various downstream tasks for better performance (Kim, 2014; Collobert et al., 2011; Pennington et al., 2014). Most of the word embedding models are trained upon <word, context> pairs in the local win- dow. Among them, word2vec gains its popu- larity by its amazing effectiveness and efficien- cy (Mikolov et al., 2013b,a). It achieves state- of-the-art results on a range of linguistic tasks with only a fraction of time compared with pre- vious techniques. A challenger of word2vec is GloVe (Pennington et al., 2014). Instead of train- ing on <word, context> pairs, GloVe directly uti- lizes word co-occurrence matrix. They claim that the change brings the improvement over word2vec on both accuracy and speed. Levy and Goldberg (2014b) further reveal that the attractive properties observed in word embeddings are not restricted to neural models such as word2vec and GloVe. They use traditional count-based method (PPMI matrix with hyper-parameter tuning) to represent word- s, and achieve comparable results with the above neural embedding models. The above models limit their information source to word co-occurrence statistics (Levy et al., 2015). To learn improved word representation- s, we extend the information source from co- occurrence of ‘word-word’ type to co-occurrence of ‘ngram-ngram’ type. The idea of using ngrams is well supported by language modeling, one of the oldest problems studied in statistical NLP. In lan- guage models, co-occurrence of words and ngrams is used to predict the next word (Kneser and Ney, 1995; Katz, 1987). Actually, the idea of …

Content truncated to 500 words