SEQˆ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression

Proceedings of NAACL-HLT 2019, pages 673–681 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 673 SEQ3: Differentiable Sequence-to-Sequence-to-Sequence Autoencoder for Unsupervised Abstractive Sentence Compression Christos Baziotis1,2, Ion Androutsopoulos2, Ioannis Konstas3, Alexandros Potamianos1 1 School of ECE, National Technical University of Athens, Athens, Greece 2 Department of Informatics, Athens University of Economics and Business, Athens, Greece 3 Interaction Lab, School of Math. and Comp. Sciences, Heriot-Watt University, Edinburgh, UK cbaziotis@mail.ntua.gr, ion@aueb.gr i.konstas@hw.ac.uk, potam@central.ntua.gr Abstract Neural sequence-to-sequence models are cur- rently the dominant approach in several nat- ural language processing tasks, but require large parallel corpora. We present a sequence- to-sequence-to-sequence autoencoder (SEQ3), consisting of two chained encoder-decoder pairs, with words used as a sequence of dis- crete latent variables. We apply the pro- posed model to unsupervised abstractive sen- tence compression, where the first and last sequences are the input and reconstructed sentences, respectively, while the middle se- quence is the compressed sentence. Constrain- ing the length of the latent word sequences forces the model to distill important infor- mation from the input. A pretrained lan- guage model, acting as a prior over the latent sequences, encourages the compressed sen- tences to be human-readable. Continuous re- laxations enable us to sample from categori- cal distributions, allowing gradient-based op- timization, unlike alternatives that rely on re- inforcement learning. The proposed model does not require parallel text-summary pairs, achieving promising results in unsupervised sentence compression on benchmark datasets. 1 Introduction Neural sequence-to-sequence models (SEQ2SEQ) perform impressively well in several natural lan- guage processing tasks, such as machine trans- lation (Sutskever et al., 2014; Bahdanau et al., 2015) or syntactic constituency parsing (Vinyals et al., 2015). However, they require massive par- allel training datasets (Koehn and Knowles, 2017). Consequently there has been extensive work on utilizing non-parallel corpora to boost the per- formance of SEQ2SEQ models (Sennrich et al., 2016; G¨ulc¸ehre et al., 2015), mostly in neural ma- chine translation where models that require ab- solutely no parallel corpora have also been pro- 𝑥���1, 𝑥���2, … , 𝑥���𝑀 𝑥1, 𝑥2, … , 𝑥𝑀��� ො𝑥1, ො𝑥2, … , ො𝑥𝑀��� Compressor (encoder-decoder) Reconstructor (encoder-decoder) Reconstruction Loss LM Prior Loss Topic Loss Figure 1: Overview of the proposed SEQ3 autoencoder. posed (Artetxe et al., 2018; Lample et al., 2018b). Unsupervised (or semi-supervised) SEQ2SEQ models have also been proposed for summariza- tion tasks with no (or small) parallel text-summary sets, including unsupervised sentence compres- sion. Current models, however, barely reach lead- N baselines (Fevry and Phang, 2018; Wang and Lee, 2018), and/or are non-differentiable (Wang and Lee, 2018; Miao and Blunsom, 2016), thus relying on reinforcement learning, which is un- stable and inefficient. By contrast, we propose a sequence-to-sequence-to-sequence autoencoder, dubbed SEQ3, that can be trained end-to-end via gradient-based optimization. SEQ3 employs dif- ferentiable approximations for sampling from cat- egorical distributions (Maddison et al., 2017; Jang et al., 2017), which have been shown to outper- form reinforcement learning (Havrylov and Titov, 2017). Therefore it is a generic framework which can be easily extended …

Content truncated to 500 words