ALIGNMEET: A Comprehensive Tool for Meeting Annotation, Alignment, and Evaluation

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 1771–1779 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 1771 ALIGNMEET: A Comprehensive Tool for Meeting Annotation, Alignment, and Evaluation Peter Pol´ak, Muskaan Singh, Anna Nedoluzhko, Ondˇrej Bojar Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics {polak,singh,nedoluzhko,bojar}@ufal.mff.cuni.cz Abstract Summarization is a challenging problem, and even more challenging is to manually create, correct, and evaluate the summaries. The severity of the problem grows when the inputs are multi-party dialogues in a meeting setup. To facilitate the research in this area, we present ALIGNMEET, a comprehensive tool for meeting annotation, alignment, and evaluation. The tool aims to provide an efficient and clear interface for fast annotation while mitigating the risk of introducing errors. Moreover, we add an evaluation mode that enables a comprehensive quality evaluation of meeting minutes. To the best of our knowledge, there is no such tool available. We release the tool as open source. It is also directly installable from PyPI. Keywords: meeting summarization, meeting minutes, minuting, annotation, evaluation 1. Introduction Meeting summarization into meeting minutes is pri- marily focused on topical coverage rather than on flu- ency or coherence. It is a challenging and tedious task, even when meeting summaries are created manually. The resulting summaries vary in the goals, style, and they are inevitably very subjective due to the human in the loop. Also, the awareness of the context of the meeting is essential to create adequate and informative summaries. 1.1. Motivation First, there is a scarcity of large-scale meeting datasets: There are a few meeting corpora, such as AMI (Mc- Cowan et al., 2005) and ICSI (Janin et al., 2003), which are rather small, on the order of a few dozens of hours each as represented in Table 1. Due to this fact, meeting summarization models are usually trained on news (Grusky et al., 2018), stories (Hermann et al., 2015), Wikipedia (Frefel, 2020; Antognini and Falt- ings, 2020), and other textual corpora, relating poorly to meetings. Second, when one tries to create such a collection or when a new meeting is to be processed, a reliable tran- script is needed, which is often impossible for the cur- rent automatic speech recognition systems (ASR). It usually requires a large amount of processing to make the transcript suitable for summarization. Third, meeting transcripts are usually long text docu- ments consisting of multi-party dialogues (see Table 1) with multiple topics. Moreover, meeting summaries are also longer compared to text summaries. The manifold structure and length of meeting transcripts and sum- maries make it difficult to traverse and follow the infor- mation for human annotators. Even training is difficult for a neural attention summarization model (Zhu et al., 2020b) with such input complexities. Finally, evaluation of meeting summarization requires immediate access to the meeting transcript and some- times even to the original sound recording to assess the quality of a particular summary point. The length of meeting …

Content truncated to 500 words