Self-Discriminative Learning for Unsupervised Document Embedding

Proceedings of NAACL-HLT 2019, pages 2465–2474 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2465 Self-Discriminative Learning for Unsupervised Document Embedding Hong-You Chen∗1, Chin-Hua Hu∗1, Leila Wehbe2, Shou-De Lin1 1Department of Computer Science and Information Engineering, National Taiwan University 2Machine Learning Department, Carnegie Mellon University {b03902128, r07922028}@ntu.edu.tw, lwehbe@cmu.edu, sdlin@csie.ntu.edu.tw Abstract Unsupervised document representation learn- ing is an important task providing pre-trained features for NLP applications. Unlike most previous work which learn the embedding based on self-prediction of the surface of text, we explicitly exploit the inter-document infor- mation and directly model the relations of doc- uments in embedding space with a discrimi- native network and a novel objective. Exten- sive experiments on both small and large pub- lic datasets show the competitiveness of the proposed method. In evaluations on standard document classification, our model has errors that are relatively 5 to 13% lower than state-of- the-art unsupervised embedding models. The reduction in error is even more pronounced in scarce label setting. 1 Introduction Rapid advance in deep methods for natural lan- guage processing has contributed to a growing need for vector representation of documents as in- put features. Applications for such vector repre- sentations include machine translation (Sutskever et al., 2014), text classification (Dai and Le, 2015), image captioning (Mao et al., 2015), multi-lingual document matching (Pham et al., 2015), question answering (Rajpurkar et al., 2016), and more. This work studies unsupervised training for encoders that can efficiently encode long paragraph of text into compact vectors to be used as pre-trained fea- tures. Existing solutions are mostly based on the assumption that a good document embedding can be learned through modeling the intra-document information by predicting the occurrence of terms inside the document itself. We argue that such an assumption might not be sufficient to obtain mean- ∗ Equally contribution. ingful a document embedding as they do not con- sider inter-document relationships. Traditional document representation models such as Bag-of-words (BoW) and TF-IDF show competitive performance in some tasks (Wang and Manning, 2012). However, these models treat words as flat tokens which may neglect other use- ful information such as word order and semantic distance. This in turn can limit the models effec- tiveness on more complex tasks that require deeper level of understanding. Further, BoW models suf- fer from high dimensionality and sparsity. This is likely to prevent them from being used as input features for downstream NLP tasks. Continuous vector representations for docu- ments are being developed. A successful thread of work is based on the distributional hypothesis, and use contextual information for context-word predictions. Similar to Word2Vec (Mikolov et al., 2013), PV (Le and Mikolov, 2014) is optimized by predicting the next words given their contexts in a document, but it is conditioned on a unique document vector. Word2Vec-based methods for computing document embeddings achieve state- of-the-art performance on document embedding. Such methods rely on one strong underlying as- sumption: it is necessary to train the document embedding to optimize the prediction of …

Content truncated to 500 words