When Hyperparameters Help: Beneficial Parameter Combinations in Distributional Semantic Models

When Hyperparameters Help: Beneficial Parameter Combinations in Distributional Semantic Models Alicia Krebs a.m.krebs@student.rug.nl Center for Mind and Brain Sciences (CIMeC), Denis Paperno denis.paperno@unitn.it University of Trento, Rovereto, Italy Abstract Distributional semantic models can pre- dict many linguistic phenomena, including word similarity, lexical ambiguity, and se- mantic priming, or even to pass TOEFL synonymy and analogy tests (Landauer and Dumais, 1997; Griffiths et al., 2007; Turney and Pantel, 2010). But what does it take to create a competitive distributional model? Levy et al. (2015) argue that the key to success lies in hyperparameter tun- ing rather than in the model’s architec- ture. More hyperparameters trivially lead to potential performance gains, but what do they actually do to improve the mod- els? Are individual hyperparameters’ con- tributions independent of each other? Or are only specific parameter combinations beneficial? To answer these questions, we perform a quantitative and qualitative eval- uation of major hyperparameters as identi- fied in previous research. 1 Introduction In a rigorous evaluation, (Baroni et al., 2014) showed that neural word embeddings such as skip- gram have an edge over traditional count-based models. However, as argued by Levy and Gold- berg (2014), the difference is not as big as it ap- pears, since skip-gram is implicitly factorizing a word-context matrix whose cells are the pointwise mutual information (PMI) of word context pairs shifted by a global constant. Levy et al. (2015) further suggest that the performance advantage of neural network based models is largely due to hy- perparameter optimization, and that the optimiza- tion of count based models can result in similar performance gains. In this paper we take this claim as the starting point. We experiment with three hyperparameters that have the greatest ef- fect on model performance according to Levy et al. (2015): subsampling, shifted PMI and context distribution smoothing. To get a more detailed pic- ture, we use a greater range of hyperparameter val- ues than in previous work, comparing all hyperpa- rameter value combinations, and perform a quali- tative analysis of their effect. 2 Hyperparameters Explored 2.1 Context Distribution Smoothing (CDS) Mikolov et al. (2013b) smoothed the original con- texts distribution raising unigram frequencies to the power of alpha. Levy and Goldberg (2015) used this technique in conjunction with PMI. PMI(w, c) = log ˆP(w, c) ˆP(w) · ˆPα(c) ˆPα(c) = #(c)α � c #(c)α After CDS, either PPMI or Shifted PPMI may be applied. We implemented CDS by raising every count to the power of α, exploring several values for α, from .25 to .95 to 1 (no smoothing). 2.2 Shifted PPMI Levy and Goldberg introduced Shifted Positive Pointwise Mutual Information (SPPMI) as an as- sociation measure more efficient than PPMI. For every word w and every context c, the SPPMI of w is the higher value between 0 and its PMI value minus the log of a constant k. PPMI(w, c) = max(log P(w, c) P(w)P(c), 0) SPPMIk(w, c) = max(PMI(w, c) − log k, 0) 2.3 Subsampling Subsampling was used by Mikolov et …

Content truncated to 500 words