Low-Rank Approximations of Second-Order Document Representations

Proceedings of the 23rd Conference on Computational Natural Language Learning, pages 634–644 Hong Kong, China, November 3-4, 2019. c⃝2019 Association for Computational Linguistics 634 Low-rank approximations of second-order document representations Jarkko Lagus Reaktor Innovations Oy, University of Helsinki jalagus@cs.helsinki.fi Janne Sinkkonen Reaktor Innovations Oy janne.sinkkonen@reaktor.fi Arto Klami University of Helsinki Department of Computer Science arto.klami@cs.helsinki.fi Abstract Document embeddings, created with methods ranging from simple heuristics to statistical and deep models, are widely applicable. Bag- of-vectors models for documents include the mean and quadratic approaches (Torki, 2018). We present evidence that quadratic statistics alone, without the mean information, can of- fer superior accuracy, fast document compar- ison, and compact document representations. In matching news articles to their comment threads, low-rank representations of only 3–4 times the size of the mean vector give most accurate matching, and in standard sentence comparison tasks, results are state of the art despite faster computation. Similarity mea- sures are discussed, and the Frobenius prod- uct implicit in the proposed method is con- trasted to Wasserstein or Bures metric from the transportation theory. We also shortly demon- strate matching of unordered word lists to doc- uments, to measure topicality or sentiment of documents. 1 Introduction Today, most computational models for natural language are based on distributional representa- tions. Words are routinely represented by word embeddings (Mikolov et al., 2013), most com- monly as fixed-dimensional real-valued vectors, such as GloVe (Pennington et al., 2014) and fast- Text (Mikolov et al., 2018). Even though there is extensive literature on using, e.g., character-level and other sub-word information (Lee et al., 2017; Radford et al., 2017; Mikolov et al., 2018) or non- Euclidean embedding spaces (Nickel and Kiela, 2017; Muzellec and Cuturi, 2018), the standard embeddings remain currently as the default build- ing block for practical tools. Most applications do not, however, care about individual words. Instead, we may be concerned about the meaning of sentences or retrieval of doc- uments, or in general, units larger than words. Distributed representations can be built for these larger units as well. Although sentences and docu- ments differ as linguistic concepts, computational models for them can be similar when they are con- sidered as sequences or even (unordered) bags of words. Document representations often build on word embeddings. Already the mean of the word vec- tors turns out to be a surprisingly good representa- tion (Wieting et al., 2015b), and accounting for the importance of words by a weighting scheme im- proves it further (Arora et al., 2016; Gupta et al., 2019). Even though the bare mean clearly ig- nores information, it is very efficient to compute. On the other end of the spectrum, document em- beddings are built with computationally extremely heavy deep learning models such as ELMo (Peters et al., 2018), ULMFiT (Howard and Ruder, 2018), and BERT (Devlin et al., 2018). Deep models pro- duce rich representations, but the amount of data and computation needed for training make them prohibitive for many applications.1 Our work falls between these two extremes. With …

Content truncated to 500 words