Data-efficient Neural Text Compression with Interactive Learning

Proceedings of NAACL-HLT 2019, pages 2543–2554 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 2543 Data-efficient Neural Text Compression with Interactive Learning Avinesh P.V.S and Christian M. Meyer Research Training Group AIPHES and UKP Lab Computer Science Department, Technische Universit¨at Darmstadt www.aiphes.tu-darmstadt.de, www.ukp.tu-darmstadt.de Abstract Neural sequence-to-sequence models have been successfully applied to text compression. However, these models were trained on huge automatically induced parallel corpora, which are only available for a few domains and tasks. In this paper, we propose a novel interactive setup to neural text compression that enables transferring a model to new domains and com- pression tasks with minimal human supervi- sion. This is achieved by employing active learning, which intelligently samples from a large pool of unlabeled data. Using this setup, we can successfully adapt a model trained on small data of 40k samples for a headline gener- ation task to a general text compression dataset at an acceptable compression quality with just 500 sampled instances annotated by a human. 1 Introduction Text compression is the task of condensing one or multiple sentences into a shorter text of a given length preserving the most important information. In natural language generation applications, such as summarization, text compression is a major step to condense the extracted important content of the source documents. But text compression can also be applied in a wide range of related applications, including the generation of headlines (Filippova et al., 2015), captions (Wubben et al., 2016), sub- titles (Vandegehinste and Pan, 2004; Luotolahti and Ginter, 2015), and the compression of text for small screens (Corston-Oliver, 2001). Neural sequence-to-sequence (Seq2Seq) mod- els have shown remarkable success in many areas of natural language processing and specifically in natural language generation tasks, including text compression (Rush et al., 2015; Filippova et al., 2015; Yu et al., 2018; Kamigaito et al., 2018). De- spite their success, Seq2Seq models have a ma- jor drawback, as they require huge parallel cor- pora with pairs of source and compressed text to be able to learn the parameters for the model. So far, the size of the training data has been propor- tional to the increase in the model’s performance (Koehn et al., 2003; Suresh, 2010), which is a ma- jor hurdle if only limited annotation capacities are available to manually produce a corpus. That is why existing research employs large-scale auto- matically extracted compression pairs, such as the first sentence and the presumably shorter headline of a news article. However, such easy-to-extract source data is only available for a few tasks, do- mains, and genres and the corresponding models do not generalize well from the task of headline generation to other text compression tasks. In this paper, we propose an interactive setup to neural text compression, which learns to compress based on user feedback acquired during training time. For the first time, we apply active learning (AL) methods to neural text compression, which greatly reduces the amount of the required train- ing data and thus yields …

Content truncated to 500 words