Summarization Based on Embedding Distributions

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1984–1989, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Summarization Based on Embedding Distributions Hayato Kobayashi Masaki Noguchi Yahoo Japan Corporation 9-7-1 Akasaka, Minato-ku, Tokyo 107-6211, Japan {hakobaya, manoguch, tyatsuka}@yahoo-corp.jp Taichi Yatsuka Abstract In this study, we consider a summariza- tion method using the document level sim- ilarity based on embeddings, or distributed representations of words, where we as- sume that an embedding of each word can represent its “meaning.” We formalize our task as the problem of maximizing a sub- modular function defined by the negative summation of the nearest neighbors’ dis- tances on embedding distributions, each of which represents a set of word embed- dings in a document. We proved the sub- modularity of our objective function and that our problem is asymptotically related to the KL-divergence between the prob- ability density functions that correspond to a document and its summary in a con- tinuous space. An experiment using a real dataset demonstrated that our method performed better than the existing method based on sentence-level similarity. 1 Introduction Document summarization aims to rephrase a doc- ument in a short form called a summary while keeping its “meaning.” In the present study, we aim to characterize the meaning of a document us- ing embeddings or distributed representations of words in the document, where an embedding of each word is represented as a real valued vector in a Euclidean space that corresponds to the word (Mikolov et al., 2013a; Mikolov et al., 2013b). Many previous studies have investigated sum- marization (Lin and Bilmes, 2010; Lin and Bilmes, 2011; Lin and Bilmes, 2012; Sipos et al., 2012; Morita et al., 2013), but to the best of our knowledge, only one (K˚ageb¨ack et al., 2014) con- sidered a direct summarization method using em- beddings, where the summarization problem was formalized as maximizing a submodular function defined by the summation of cosine similarities based on sentence embeddings. Essentially, this method assumes linear meanings since the objec- tive function is characterized by the summation of sentence-level similarities. However, this assump- tion is not always valid in real documents, and thus there may be a better combination of two other sentences than the best and second best sentences in terms of similarity in a document. In this study, we consider a summarization method based on document-level similarity, where we assume the non-linearity of meanings. First, we examine an objective function defined by a co- sine similarity based on document embeddings in- stead of sentence embeddings. Unfortunately, in contrast to our intuition, this similarity is not sub- modular, which we disprove later. Thus, we pro- pose a valid submodular function based on em- bedding distributions, each of which represents a set of word embeddings in a document, as the document-level similarity. Our objective func- tion is calculated based on the nearest neighbors’ distances on embedding distributions, which can be proved to be asymptotically related to KL- divergence in a continuous space. …

Content truncated to 500 words