An Empirical Comparison of off-the-shelve Semantic Similarity methods for down-streaming Meeting Similarity

An Empirical Comparison of Semantic Similarity Methods for Analyzing down-streaming Automatic Minuting task Aditya Upadhyay&, Aakash Bhatnagar1, Nidhir Bhavsar$, Muskaan Singh# and Petr Motlicek# & CSED, Thapar Institute of Engineering and Technology, India 1 Boston University, Boston, Massachusetts $University of Potsdam, Potsdam, Germany # Speech and Audio Processing Group, IDIAP Research Institute, Martigny, Switzerland adityaupadhyay1912@gmail.com, aakash07@bu.edu, bhavsar@uni-potsdam.de, (msingh,petr.motlicek)@idiap.ch Abstract Automatic Minuting consists of automati- cally creating minutes from multiparty meet- ing transcripts. In this paper, we solve two rel- evant problems of this domain (1) given a pair of meeting transcript and minute, the task is to identify whether the minutes belongs to the transcript. (2) given a pair of minutes, the task is to identify whether the two minutes belong to the same or different meetings. These chal- lenging problems are important as we want to uncover how minutes created by two differ- ent persons for the same meeting may differ in content and coverage. The proposed sys- tem leverage off-the-shelf semantic similarity techniques which provides with a score of sim- ilarity indicating a measure of how close the two text are to each other in meaning. In per- formance analysis, we broadly formulate three categories with the best performers in each (1) in lexical summarization DOC2VEC (2) in machine learning (3) in deep transformer architectures. We evaluate each of our pro- posed approaches on the basis of Accuracy. For lexical summarization, Doc2Vec achieves 90% and 51% accuracy, in machine learning, random forest achieves 91% and 85% accu- racy and in deep learning, STSB-BERT-Large achieves 94% and 81% accuracy in transcript- minutes and minutes-minutes respectively. 1 Introduction Due to differences in the style of minuting there are two important challenges,(i) identify if the minutes and the transcript are from the same meet- ing. (ii) identify if two minutes are from the same meeting (which are taken by different note takers). In this paper, we focus to solve this novel problem and see to what extent this decision can be carried out automatically. The novelty of our research is to examine the subjectivity associated with the minut- ing exercise. Minuting is a challenging task [1], and even more difficult is identify meetings simi- larity on similar topics with (1) similarity of dis- cussed content and anchor points like named enti- ties e.g. in recurring meetings of the same project on the one hand, and the differences in the style of minuting on the other hand. (2) some minutes do not capture the central points in the meeting be- cause the external scribes did not understand the context correctly and created minutes which miss significant issues discussed in the meeting or are simply too short. 2 Background Semantic Similarity is a Natural Language Pro- cessing (NLP) task that consists of measuring the similarity between two texts in a quantitative man- ner. Measuring semantic similarity between two texts serves an important role in various NLP tasks such as information retrieval, text summa- rization, text classification, essay evaluation, ma- chine translation, and question …

Content truncated to 500 words