Asking It All: Generating Contextualized Questions for any Semantic Role

Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 117–132 November 7–11, 2021. c⃝2021 Association for Computational Linguistics 117 Decision-Focused Summarization Chao-Chun Hsu University of Chicago chaochunh@uchicago.edu Chenhao Tan University of Chicago chenhao@uchicago.edu Abstract Relevance in summarization is typically de- fined based on textual information alone, with- out incorporating insights about a particular decision. As a result, to support risk analy- sis of pancreatic cancer, summaries of medical notes may include irrelevant information such as a knee injury. We propose a novel prob- lem, decision-focused summarization, where the goal is to summarize relevant information for a decision. We leverage a predictive model that makes the decision based on the full text to provide valuable insights on how a decision can be inferred from text. To build a summary, we then select representative sentences that lead to similar model decisions as using the full text while accounting for textual non-redundancy. To evaluate our method (DecSum), we build a testbed where the task is to summarize the first ten reviews of a restaurant in support of pre- dicting its future rating on Yelp. DecSum sub- stantially outperforms text-only summarization methods and model-based explanation meth- ods in decision faithfulness and representative- ness. We further demonstrate that DecSum is the only method that enables humans to out- perform random chance in predicting which restaurant will be better rated in the future. 1 Introduction Human decision making often requires making sense of a large amount of information. For in- stance, doctors go through a myriad of medical notes to determine the risk of pancreatic cancer, and investors need to decide whether a stock price will increase based on hundreds of analyst reports. In these cases, summarization can potentially support human decision making by identifying the most relevant information for these decisions (Demner- Fushman et al., 2009; Workman et al., 2012). Ideally, decision-focused summarization should incorporate insights about how decisions can be inferred from text. However, typical summariza- tion methods in NLP define relevance based on 1 2 3 4 5 Model prediction 0.0 0.2 0.4 0.6 0.8 Density DecSum PreSumm Integrated Gradient Figure 1: Illustration of the selected sentences by dif- ferent methods on the distribution of model predictions on all individual sentences. Our method (DecSum) cov- ers the full distribution, while PreSumm, a text-only summarization method, concentrates on the right side, and integrated gradients, a model-explanation method, misses the middle part. the textual information exclusively. An example desideratum is textual non-redundancy (Carbonell and Goldstein, 1998), which encourages the sum- maries to cover diverse information in the input documents. Fully optimizing this text-only crite- rion can be counter-productive for decision making: information about a knee injury does not really help understand the risk of pancreatic cancer, and the disclaimers in financial analysts may not be the most relevant for investment decisions. In this work, we investigate the potential of leveraging a supervised decision model for extrac- tive decision-focused summarization. A predictive model that learns to make a decision given the full …

Content truncated to 500 words