AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2508 - 2520 July 10-15, 2022 ©2022 Association for Computational Linguistics AnswerSumm: A Manually-Curated Dataset and Pipeline for Answer Summarization Alexander R. Fabbri† ∗ Xiaojian Wu‡ Srini Iyer‡ Haoran Li‡ Mona Diab‡ † Yale University ‡ Facebook AI afabbri@salesforce.com {xiaojianwu,sviyer,aimeeli,mdiab}@fb.com Abstract Community Question Answering (CQA) fora such as Stack Overflow and Yahoo! Answers contain a rich resource of answers to a wide range of community-based questions. Each question thread can receive a large number of answers with different perspectives. One goal of answer summarization is to produce a summary that reflects the range of answer perspectives. A major obstacle for this task is the absence of a dataset to provide super- vision for producing such summaries. Recent works propose heuristics to create such data, but these are often noisy and do not cover all answer perspectives present. This work intro- duces a novel dataset of 4,631 CQA threads for answer summarization curated by profes- sional linguists. Our pipeline gathers annota- tions for all subtasks of answer summarization, including relevant answer sentence selection, grouping these sentences based on perspec- tives, summarizing each perspective, and pro- ducing an overall summary. We analyze and benchmark state-of-the-art models on these subtasks and introduce a novel unsupervised approach for multi-perspective data augmen- tation that boosts summarization performance according to automatic evaluation. Finally, we propose reinforcement learning rewards to im- prove factual consistency and answer coverage and analyze areas for improvement. 1 Introduction In a world of information overload and the ubiquity of discussion fora, there is a need for text summa- rization as a means of distilling relevant informa- tion into a concise form. The problem is even more pertinent for question answering within the context of Community Question Answering (CQA) fora, where a person poses a question and can get an abundance of answers to sift through. Ideally, an answer summary should cover the multiple perspec- tives found in the answers, where available. Table 1 ∗Author is currently at Salesforce AI Research. Question: I recently relocated to USA and have no Credit Score. Is Secure Credit Card is the only option for me to start building my credit score? Also please recommend which other credit cards are available for people like me to build credit score Answer 1: If you have an AMEX from another country, you can get an AMEX in the US. American Express has a separate system that is not as strongly country-dependent as, say, VISA and MasterCard... Answer 2: Secured credit cards are usually not very cost effective for building credit. Find a local credit union, of medium to large size. A credit union is like a bank, but operates under slightly different rules, and is non-profit... Answer 3: If you have had an American Express card abroad, you can try and get a US Amex... Answer 4: If the country you came from has an HSBC, you can ask …

Content truncated to 500 words