MIDAS: A Dialog Act Annotation Scheme for Open Domain HumanMachine Spoken Conversations

Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics, pages 1103–1120 April 19 - 23, 2021. ©2021 Association for Computational Linguistics 1103 MIDAS: A Dialog Act Annotation Scheme for Open Domain Human Machine Spoken Conversations Dian Yu University of California, Davis dianyu@ucdavis.edu Zhou Yu Columbia University zhouyu@cs.columbia.edu Abstract Dialog act prediction in open-domain con- versations is an essential language compre- hension task for both dialog system build- ing and discourse analysis. Previous dialog act schemes, such as SWBD-DAMSL, are de- signed mainly for discourse analysis in human- human conversations. In this paper, we present a dialog act annotation scheme, MIDAS (Ma- chine Interaction Dialog Act Scheme), tar- geted at open-domain human-machine conver- sations. MIDAS is designed to assist machines to improve their ability to understand human partners. MIDAS has a hierarchical struc- ture and supports multi-label annotations. We collected and annotated a large open-domain human-machine spoken conversation dataset (consisting of 24K utterances). To validate our scheme, we leveraged transfer learning meth- ods to train a multi-label dialog act prediction model and reached an F1 score of 0.79.1 1 Introduction Human-machine conversations have different dy- namics compared to human-human conversations due to the power imbalance between humans and machines in conversational settings. Such differ- ences include content and voice quality (Hill et al., 2015). For instance, humans tend to use a more authoritative voice when they talk to a machine. We found that “commands” account for about 9% of utterances in a human-machine social conver- sation corpus we collected, while a similar dialog act, “Action-Directive”, only accounts for 0.4% in a human-human conversation corpus (Switch- board Dialog Act Corpus (SwDA) (Jurafsky et al., 1997)). Moreover, a human-machine dialog act scheme is not simply used to understand the dia- log flow but also to help dialog systems plan their 1Code, data, and trained models are available at https: //github.com/DianDYu/MIDAS_dialog_act Scheme #labels context multi-label dataset DAMSL 44    TOPIC 14    ISO 88 ?   MIDAS 23    Table 1: Comparison of different major dialog act schemes. TOPIC refers to the dialog act designed for topic modeling from Khatri et al. (2018). “?” indicates that guideline on context using is not clear (Mezza et al., 2018) next steps. Therefore, a dialog act scheme for spo- ken dialog systems must capture semantic infor- mation necessary for fine-grained dialog planning. More importantly, a useful human-machine dialog act scheme should incorporate dynamics of real- istic settings such as real-time automatic speech recognition (ASR) outputs. However, previously available dialog act schemes are all designed for manual dialog transcriptions, which are very differ- ent from ASR outputs. For instance, ASR outputs are noisy and may not contain punctuation. To test if a dialog act predictor trained on manual tran- scriptions would generalize to ASR outputs, we trained a model using the SwDA dataset annotated with the SWBD-DAMSL scheme (Jurafsky et al., 1997). We tested the model on the ASR output of our open-domain human-machine dialog …

Content truncated to 500 words