Word Embeddings based on Fixed-Size Ordinally Forgetting Encoding

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 310–315 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Word Embeddings based on Fixed-Size Ordinally Forgetting Encoding Joseph Sanu†, Mingbin Xu†, Hui Jiang† and Quan Liu‡ †Department of Electrical Engineering and Computer Science, York University, 4700 Keele Street, Toronto, Ontario, M3J 1P3, Canada ‡Department of EEIS, University of Science and Technology of China, Hefei, China cse83186@cse.yorku.ca, xmb@cse.yorku.ca, hj@cse.yorku.ca, quanliu@mail.ustc.edu.cn Abstract In this paper, we propose to learn word embeddings based on the recent fixed- size ordinally forgetting encoding (FOFE) method, which can almost uniquely en- code any variable-length sequence into a fixed-size representation. We use FOFE to fully encode the left and right con- text of each word in a corpus to con- struct a novel word-context matrix, which is further weighted and factorized using truncated SVD to generate low-dimension word embedding vectors. We have eval- uated this alternative method in encoding word-context statistics and show the new FOFE method has a notable effect on the resulting word embeddings. Experimen- tal results on several popular word similar- ity tasks have demonstrated that the pro- posed method outperforms many recently popular neural prediction methods as well as the conventional SVD models that use canonical count based techniques to gen- erate word context matrices. 1 Introduction Low dimensional vectors as word representations are very popular in NLP tasks such as inferring semantic similarity and relatedness. Most of these representations are based on either matrix factor- ization or context sampling described by (Baroni et al., 2014) as count or predict models. The ba- sis for both models is the distributional hypoth- esis (Harris, 1954), which states that words that appear in similar contexts have similar meaning. Traditional context representations have been ob- tained by capturing co-occurrences of words from a fixed-size window relative to the focus word. This representation however does not encompass the entirety of the context surrounding the focus word. Therefore, the distributional hypothesis is not being taken advantage of to the fullest extent. In this work, we seek to capture these contexts through the fixed-size ordinally forgetting encod- ing (FOFE) method, recently proposed in (Zhang et al., 2015b). In addition to just capturing word co-occurrences, we attempt to use the FOFE to encode the full contexts of each focus word, in- cluding the order information of the context se- quences. We believe the full encoding of con- texts can enhance the resulting word embedding vectors, derived by factoring the corresponding word-context matrix. As argued in (Zhang et al., 2015b), the FOFE method can almost uniquely en- code discrete sequences of varying lengths into a fixed-size code, and this encoding method was used to address the challenges of a limited size window when using deep neural networks for lan- guage modeling. The resulting algorithm fulfills the needs of keeping long term dependency while being fast. The word order in a sequence is mod- eled by FOFE using an ordinally-forgetting mech- anism which encodes each position of …

Content truncated to 500 words