Cyberbullying Classifiers are Sensitive to Model-Agnostic Perturbations

Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), pages 298–304 Marseille, 20-25 June 2022 © European Language Resources Association (ELRA), licensed under CC-BY-NC-4.0 298 Evaluating the Effects of Embedding with Speaker Identity Information in Dialogue Summarization Yuji Naraki, Tetsuya Sakai, Yoshihiko Hayashi Faculty of Science and Engineering, Waseda University Tokyo, 169-8555 JAPAN yuji.1277@akane.waseda.jp, tetsuyasakai@acm.org, yshk.hayashi@aoni.waseda.jp Abstract Automatic dialogue summarization is a task used to succinctly summarize a dialogue transcript while correctly linking the speakers and their speech, which distinguishes this task from a conventional document summarization. To address this issue and reduce the “who said what”-related errors in a summary, we propose embedding the speaker identity information in the input embedding into the dialogue transcript encoder. Unlike the speaker embedding proposed by Gu et al. (2020), our proposal takes into account the informativeness of position embedding. By experimentally comparing several embedding methods, we confirmed the ROUGE and human evaluation scores of the generated summaries were substantially increased by embedding speaker information at the less informative part of the fixed position embedding with sinusoidal functions. Keywords: summarization, dialogue, embedding 1. Introduction There has been an increasing demand for automatic di- alogue summarization in real-world applications, for example, in summarizing interactions in customer ser- vice centers and hospitals. An example of data for a di- alogue summarization task is shown in Figure 1. How- ever, for this task, there has been little research con- ducted on a dialogue summarization owing to a lack of high-quality datasets. Specifically, until SAMSum (Gliwa et al., 2019) has been available, there were few studies on dialogue-specific deep learning methods be- cause there are no suitable or sufficiently large public datasets for the training of deep neural network (DNN) models. As the major difference between document and dialogue summarizations, the connection between speakers and their speech must be correctly captured by the model. Therefore, we focus on speaker iden- tity information contained in dialogues and propose an embedding of speaker identity information as one of the input embeddings of Transformer-based models (Vaswani et al., 2017). More concretely, our proposed embedding is added only to the less informative parts of the position embedding. Experimental results demon- strate that the proposed method improves the conver- gence of the model in training and increases the av- erage ROUGE scores of the generated summaries in comparison to existing methods of document and dia- logue summarization. 2. Related Work Abstractive Summarization. Liu and Lapata (2019) proposed BERTSumAbs, which apply BERT to an ab- stractive document summarization. However, the pre- training in BERTSumAbs is only applied to the en- coder. Zhang et al. (2020) proposed PEGASUS, an ab- stractive summarization model that uses a pre-training approach called a gap sentence generation (GSG). GSG Figure 1: Example of dialogue and summary of SAMSum dataset (Gliwa et al., 2019) . A dialogue is a list of name-speech pairs, and a summary contains the names of the speakers in the dialogue. enables the decoder to be pre-trained, making it possi- ble to …

Content truncated to 500 words