EmailSum: Abstractive Email Thread Summarization

Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6895–6909 August 1–6, 2021. ©2021 Association for Computational Linguistics 6895 EMAILSUM: Abstractive Email Thread Summarization Shiyue Zhang♠ Asli Celikyilmaz♥ Jianfeng Gao♣ Mohit Bansal♠ ♠UNC Chapel Hill ♥Facebook AI Research ♣Microsoft Research {shiyue, mbansal}@cs.unc.edu aslic@fb.com jfgao@microsoft.com Abstract Recent years have brought about an interest in the challenging task of summarizing conver- sation threads (meetings, online discussions, etc.). Such summaries help analysis of the long text to quickly catch up with the deci- sions made and thus improve our work or communication efficiency. To spur research in thread summarization, we have developed an abstractive Email Thread Summarization (EMAILSUM) dataset, which contains human- annotated short (<30 words) and long (<100 words) summaries of 2,549 email threads (each containing 3 to 10 emails) over a wide variety of topics. We perform a comprehensive empirical study to explore different summa- rization techniques (including extractive and abstractive methods, single-document and hi- erarchical models, as well as transfer and semi- supervised learning) and conduct human eval- uations on both short and long summary gen- eration tasks. Our results reveal the key chal- lenges of current abstractive summarization models in this task, such as understanding the sender’s intent and identifying the roles of sender and receiver. Furthermore, we find that widely used automatic evaluation metrics (ROUGE, BERTScore) are weakly correlated with human judgments on this email thread summarization task. Hence, we emphasize the importance of human evaluation and the devel- opment of better metrics by the community.1 1 Introduction As one of the major natural language generation tasks, automatic summarization has been studied for decades. Most research efforts were focused on single-document summarization tasks, e.g., news document summarization (Hermann et al., 2015; Narayan et al., 2018). However, living in an in- formation era, we are facing with diverse content 1Our code and summary data have been made available at: https://github.com/ZhangShiyue/EmailSum Email Thread: Subject: lunch this week Susan: All, Regarding our lunch this week to celebrate the one year anniversaries for Michelle & David, and Mark’s birthday, I have a request to make it Wednesday instead of Tuesday. Does anyone have an objection to this? Susan David: I have another lunch engagement Wed, but I will skip it if everyone else wants to move our lunch. David Tamra: Susan, Wednesday works out better for me as well. I have a doctor’s appointment tomorrow during lunch. Tamra Short Summary: Susan emails everyone about an anniversary and offers to change the date. David says he is busy but is willing to go with the majority. Tamra agrees with Susan’s date. Long Summary: Susan emails everyone about a lunch to celebrate a one year anniversary as well as Mark’s birthday. She says she would change the date to a different day. David says he is busy that day with his own appointment but is willing to go with the majority and cancel that appointment to make this one. Tamra …

Content truncated to 500 words