Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6331–6338 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 6331 Encouraging Paragraph Embeddings to Remember Sentence Identity Improves Classification Tu Vu and Mohit Iyyer College of Information and Computer Sciences University of Massachusetts Amherst {tuvu,miyyer}@cs.umass.edu Abstract While paragraph embedding models are re- markably effective for downstream classifica- tion tasks, what they learn and encode into a single vector remains opaque. In this paper, we investigate a state-of-the-art paragraph embed- ding method proposed by Zhang et al. (2017) and discover that it cannot reliably tell whether a given sentence occurs in the input paragraph or not. We formulate a sentence content task to probe for this basic linguistic property and find that even a much simpler bag-of-words method has no trouble solving it. This re- sult motivates us to replace the reconstruction- based objective of Zhang et al. (2017) with our sentence content probe objective in a semi- supervised setting. Despite its simplicity, our objective improves over paragraph reconstruc- tion in terms of (1) downstream classification accuracies on benchmark datasets, (2) faster training, and (3) better generalization ability.1 1 Introduction Methods that embed a paragraph into a sin- gle vector have been successfully integrated into many NLP applications, including text classifica- tion (Zhang et al., 2017), document retrieval (Le and Mikolov, 2014), and semantic similarity and relatedness (Dai et al., 2015; Chen, 2017). How- ever, downstream performance provides little in- sight into the kinds of linguistic properties that are encoded by these embeddings. Inspired by the growing body of work on sentence-level linguis- tic probe tasks (Adi et al., 2017; Conneau et al., 2018), we set out to evaluate a state-of-the-art paragraph embedding method using a probe task to measure how well it encodes the identity of the sentences within a paragraph. We discover that the method falls short of capturing this basic prop- erty, and that implementing a simple objective to 1Source code and data are available at https://github.com/ tuvuumass/SCoPE. fix this issue improves classification performance, training speed, and generalization ability. We specifically investigate the paragraph em- bedding method of Zhang et al. (2017), which consists of a CNN-based encoder-decoder model (Sutskever et al., 2014) paired with a re- construction objective to learn powerful paragraph embeddings that are capable of accurately recon- structing long paragraphs. This model signifi- cantly improves downstream classification accura- cies, outperforming LSTM-based alternatives (Li et al., 2015). How well do these embeddings encode whether or not a given sentence appears in the para- graph? Conneau et al. (2018) show that such iden- tity information is correlated with performance on downstream sentence-level tasks. We thus design a probe task to measure the extent to which this sentence content property is captured in a para- graph embedding. Surprisingly, our experiments (Section 2) reveal that despite its impressive down- stream performance, the model of Zhang et al. (2017) substantially underperforms a simple bag- of-words model on our sentence content probe. Given …

Content truncated to 500 words