Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2190–2196 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 2190 Keep Meeting Summaries on Topic: Abstractive Multi-Modal Meeting Summarization Manling Li1, Lingyu Zhang2, Heng Ji1, Richard J. Radke2 1 Department of Computer Science 2 Department of Electrical, Computer, and Systems Engineering Rensselaer Polytechnic Institute 1{lim22,jih}@rpi.edu, 2{zhangl34@rpi.edu, rjradke@ecse.rpi.edu} Abstract Transcripts of natural, multi-person meetings differ significantly from documents like news articles, which can make Natural Language Generation models generate unfocused sum- maries. We develop an abstractive meeting summarizer from both videos and audios of meeting recordings. Specifically, we propose a multi-modal hierarchical attention mechanism across three levels: topic segment, utterance and word. To narrow down the focus into topically-relevant segments, we jointly model topic segmentation and summarization. In ad- dition to traditional textual features, we intro- duce new multi-modal features derived from visual focus of attention, based on the as- sumption that an utterance is more important if its speaker receives more attention. Exper- iments show that our model significantly out- performs the state-of-the-art with both BLEU and ROUGE measures. 1 Introduction Automatic meeting summarization is valuable, es- pecially if it takes advantage of multi-modal sens- ing of the meeting environment, such as micro- phones to capture speech and cameras to capture each participant’s head pose and eye gaze. Tradi- tional extractive summarization methods based on selecting and reordering salient words tend to pro- duce summaries that are not natural and incoher- ent. Although state-of-the-art work (Shang et al., 2018) employs WordNet (Miller, 1995) to make summaries more abstractive, the quality is still far from those produced by humans, as shown in Ta- ble 1. Moreover, these methods tend to have lim- ited content coverage by selecting salient words. On the other hand, recent years have wit- nessed the success of Natural Language Genera- tion (NLG) models to generate abstractive sum- maries. Since human-written summaries tend to mention the exact given keywords without para- phrasing, the copy mechanism proposed by a Pointer Generator Network (PGN) (See et al., 2017) naturally fits this task. Apart from gener- ating words from a fixed vocabulary, it also copies the words from the input. However, transcripts of multi-person meetings widely differ from tra- ditional documents. Instead of grammatical, well- segmented sentences, the input is often composed of ill-formed utterances. Therefore, NLG models can easily lose focus. For example, in Table 1, PGN fails to capture the keywords remote control, trendy and user-friendly. Therefore, we propose a multi-modal hierarchi- cal attention mechanism across topic segments, ut- terances, and words. We learn topic segmentation as an auxiliary task and limit the attention within each segment. Our approach mimics human sum- marization methods by segmenting first and then summarizing each segment. To locate key utter- ances, we propose that the rich multi-modal data from recording the meeting environment, espe- cially cameras facing each participant, can pro- vide speaker interaction and participant feedback to discover salient utterances. One typical interac- tion …

Content truncated to 500 words