Random Positive-Only Projections: PPMI-Enabled Incremental Semantic Space Construction

Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics (*SEM 2016), pages 189–198, Berlin, Germany, August 11-12, 2016. Random Positive-Only Projections: PPMI-Enabled Incremental Semantic Space Construction Behrang QasemiZadeh DFG SFB 991 Heinrich-Heine-Universit¨at D¨usseldorf D¨usseldorf, Germany zadeh@phil.hhu.de Laura Kallmeyer DFG SFB 991 Heinrich-Heine-Universit¨at D¨usseldorf D¨usseldorf, Germany kallmeyer@phil.hhu.de Abstract We introduce positive-only projection (PoP), a new algorithm for constructing semantic spaces and word embeddings. The PoP method employs random projec- tions. Hence, it is highly scalable and com- putationally efficient. In contrast to pre- vious methods that use random projec- tion matrices R with the expected value of 0 (i.e., E(R) = 0), the proposed method uses R with E(R) > 0. We use Kendall’s τb correlation to compute vector similarities in the resulting non-Gaussian spaces. Most importantly, since E(R) > 0, weighting methods such as positive pointwise mutual information (PPMI) can be applied to PoP-constructed spaces after their construction for efficiently transfer- ring PoP embeddings onto spaces that are discriminative for semantic similarity as- sessments. Our PoP-constructed models, combined with PPMI, achieve an average score of 0.75 in the MEN relatedness test, which is comparable to results obtained by state-of-the-art algorithms. 1 Introduction The development of data-driven methods of nat- ural language processing starts with an educated guess, a distributional hypothesis: We assume that some properties of linguistic entities can be modelled by ‘some statistical’ observations in lan- guage data. In the second step, this statistical in- formation (which is determined by the hypothe- sis) is collected and represented in a mathemati- cal framework. In the third step, tools provided by the chosen mathematical framework are used to implement a similarity-based logic to iden- tify linguistic structures, and/or to verify the pro- posed hypothesis. Harris’s distributional hypothe- sis (Harris, 1954) is a well-known example of step one that states that meanings of words correlate with the environment in which the words appear. Vector space models and η-normed-based similar- ity measures are notable examples of steps two and three, respectively (i.e., word space models or word embeddings). However, as pointed out for instance by Ba- roni et al. (2014), the count-based models resulting from the steps two and three are not discrimina- tive enough to achieve satisfactory results; instead, predictive models are required. To this end, an ad- ditional transformation step is often added. Tur- ney and Pantel (2010) describe this extra step as a combination of weighting and dimensionality re- duction.1 This transformation from count-based to predictive models can be implemented simply via a collection of rules of thumb (such as frequency threshold to filter out highly frequent and/or rare context elements), and/or it can involve more so- phisticated mathematical transformations, such as converting raw counts to probabilities and using matrix factorization techniques. Likewise, by ex- ploiting the large amounts of computational power available nowadays, this transformation can be achieved via neural word embedding techniques (Mikolov et al., 2013; Levy and Goldberg, 2014). To a large extent, the need for such transfor- mations arises from the heavy-tailed distributions that …

Content truncated to 500 words