Query-Based Single Document Summarization Using an Ensemble Noisy Auto-Encoder

Query-Based Single Document Summarization Using an Ensemble Noisy Auto-Encoder Mahmood Yousefi Azar, Kairit Sirts, Diego Moll´a Aliod and Len Hamey Department of Computing Macquarie University, Australia mahmood.yousefiazar@students.mq.edu.au, {kairit.sirts, diego.molla-aliod, len.hamey}@mq.edu.au Abstract In this paper we use a deep auto-encoder for extractive query-based summarization. We experiment with different input repre- sentations in order to overcome the prob- lems stemming from sparse inputs charac- teristic to linguistic data. In particular, we propose constructing a local vocabulary for each document and adding a small random noise to the input. Also, we propose us- ing inputs with added noise in an Ensem- ble Noisy Auto-Encoder (ENAE) that com- bines the top ranked sentences from mul- tiple runs on the same input with different added noise. We test our model on a pub- licly available email dataset that is specifi- cally designed for text summarization. We show that although an auto-encoder can be a quite effective summarizer, adding noise to the input and running a noisy ensemble can make improvements. 1 Introduction Recently, deep neural networks have gained pop- ularity in a wide variety of applications, in partic- ular, they have been successfully applied to vari- ous natural language processing (NLP) tasks (Col- lobert et al., 2011; Srivastava and Salakhutdinov, 2012). In this paper we apply a deep neural net- work to query-based extractive summarization task. Our model uses a deep auto-encoder (AE) (Hinton and Salakhutdinov, 2006) to learn the latent repre- sentations for both the query and the sentences in the document and then uses a ranking function to choose certain number of sentences to compose the summary. Typically, automatic text summarization systems use sparse input representations such as tf-idf. How- ever, sparse inputs can be problematic in neural network training and they may make the training slow. We propose two techniques for reducing sparsity. First, we compose for each document a local vocabulary which is then used to construct the input representations for sentences in that doc- ument. Second, we add small random noise to the inputs. This technique is similar to the denoising auto-encoders (Vincent et al., 2008). However, the denoising AE adds noise to the inputs only during training, while during test time we also add noise to input. An additional advantage of adding noise dur- ing testing is that we can use the same input with different added noise in an ensemble. Typically, an ensemble learner needs to learn several differ- ent models. However, the Ensemble Noisy Auto- Encoder (ENAE) proposed in this paper only needs to train one model and the ensemble is created from applying the model to the same input several times, each time with different added noise. Text summarization can play an important role in different application domains. For instance, when performing a search in the mailbox according to a keyword, the user could be shown short summaries of the relevant emails. This is especially attractive when using a smart phone with a small screen. We also evaluate our model on a publicly available email dataset …

Content truncated to 500 words