A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations

Proceedings of the 28th International Conference on Computational Linguistics, pages 4969–4980 Barcelona, Spain (Online), December 8-13, 2020 4969 A Large-Scale Corpus of E-mail Conversations with Standard and Two-Level Dialogue Act Annotations Motoki Taniguchi†,‡, Yoshihiro Ueda†, Tomoki Taniguchi† and Tomoko Ohkuma† {motoki.taniguchi, ueda.yoshihiro, taniguchi.tomoki, ohkuma.tomoko}@fujixerox.co.jp † Fuji Xerox Co., Ltd. ‡ Tokyo Metropolitan University Abstract We present a large-scale corpus of e-mail conversations with domain-agnostic and two-level di- alogue act (DA) annotations towards the goal of a better understanding of asynchronous conver- sations. We annotate over 6,000 messages and 35,000 sentences from more than 2,000 threads. For a domain-independent and application-independent DA annotations, we choose ISO stan- dard 24617-2 as the annotation scheme. To assess the difficulty of DA recognition on our cor- pus, we evaluate several models, including a pre-trained contextual representation model, as our baselines. The experimental results show that BERT outperforms other neural network mod- els, including previous state-of-the-art models, but falls short of a human performance. We also demonstrate that DA tags of two-level granularity enable a DA recognition model to learn effi- ciently by using multi-task learning. An evaluation of a model trained on our corpus against other domains of asynchronous conversation reveals the domain independence of our DA annotations. 1 Introduction The recent growth of textual communication media such as e-mails and online forums has led to the great demand for techniques enabling the automatic analysis of conversational structures from texts for information retrieval and intelligent assistance. Dialogue acts (DAs), which are also known as “speech acts” in some studies, are one of such conversational structures. The DAs of text in conversations are defined as having communicative functions, such as asking questions, requesting some information, and offering suggestions. Analyzing the DAs in conversations helps many downstream applications, including summarization (Bhatia et al., 2014; Oya and Carenini, 2014), question answering (Hong and Davison, 2009), and conversational agents (Peskov et al., 2019). Conversations fall into synchronous conversations (e.g., phone calls and meetings) and asynchronous conversations (e.g., e-mails and online forums). In a synchronous conversation all participants engage at the same time with all participants, whereas in an asynchronous conversation participants interact with each other at different times. As is well known, the asynchronous properties make the conversational flow of asynchronous conversations different from those of synchronous conversations (Joty et al., 2013; Louis and Cohen, 2015). Topics in asynchronous conversations are often interleaved and not contiguous sequence. This complexity in a conversational flow makes DA recognition in asynchronous conversations a challenging task compared to synchronous conversations, particularly when the thread structure (reply relations) is missing. The existing corpora for DA recognition in asynchronous conversations have some of the shortcom- ings: (i) Scale: Whereas large-scale corpora such as the Meeting Recorder Dialog Act (MRDA) (Shriberg et al., 2004) and the Switchboard Discourse Annotation and Markup System of Labeling (SWBD) (Juraf- sky et al., 1997) are available in synchronous conversation, most of the available corpora in asynchronous conversations are limited to a few thousands messages or sentences, and are insufficient …

Content truncated to 500 words