Doc2hash: Learning Discrete Latent variables for Documents Retrieval

Proceedings of NAACL-HLT 2019, pages 2235–2240 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2235 Doc2hash: Learning Discrete Latent Variables for Document Retrieval Yifei Zhang Vector Lab JD Finance zhangyifei11@jd.com Hao Zhu Vector Lab JD Finance zhuhao6@jd.com Abstract Learning to hash via generative models has be- come a powerful paradigm for fast similarity search in documents retrieval. To get binary representation (i.e., hash codes), the discrete distribution prior (i.e., Bernoulli Distribution) is applied to train the variational autoencoder (VAE). However, the discrete stochastic layer is usually incompatible with the backpropaga- tion in the training stage and thus causes a gra- dient flow problem. In this paper, we propose a method, Doc2hash, that solves the gradient flow problem of the discrete stochastic layer by using continuous relaxation on priors, and trains the generative model in an end-to-end manner to generate hash codes. In qualita- tive and quantitative experiments, we show the proposed model outperforms other state of the art methods. 1 Introduction A popular theme for deep learning is that of rep- resentation learning, whose goal is to use exist- ing data to learn a compact and meaningful repre- sentation when building classifiers or other predic- tors. Learning continuous representation has been achieving a great success in various NLP tasks, in- cluding text classification, word and sentence rep- resentation (Mikolov et al., 2013; Le and Mikolov, 2014), yet learning discrete representation is po- tentially more suitable for tasks we are interested in such as learning to hash. Hashing serves as a fast solution for similarity search also called approximate nearest neighbor search. In document retrieval, semantic hashing is the strategy that turn the document into binary codes (hash codes) which capture semantic infor- mation. One can use a query (a document) to re- trieve similar documents by calculating the ham- ming distances between their hash codes. Given the fast calculation process and the efficient stor- age property (one modern PC can execute millions Figure 1: The end-to-end semantic hashing frame- work: Training the cycle x −→ z −→ x to ob- tain the latent variables z. The one hot representa- tion of discrete latent variables z has been relaxed to {(0.05, 0.95), (0.92, 0.08), (0.99, 0.01)} by the tempt- ing softmax. Hash codes zh = {1, 0, 0} is obtained via taking the arg max of z. Y is the label/tag used in the supervised hashing of hamming distance computations in just a few milliseconds). This semantic hashing strategy is very attractive. Inspired by the recent success of modeling latent variables via generative models in solv- ing various NLP problems (van den Oord et al., 2017; Semeniuta et al., 2017; Bowman et al., 2015), Some approaches obtain binary codes of documents from a generative perspective via pa- rameterizing models using neural networks and stochastic optimization using gradient-based tech- niques: (Chaidaroon and Fang, 2017) designed a two-stage training procedures to generate hash codes with variational autoencoder: (i) it first infers continuous representations of text through VAE with …

Content truncated to 500 words