Transfer of Corpus-Specific Dialogue Act Annotation to ISO Standard: Is it worth it?

Transfer of Corpus-Specific Dialogue Act Annotation to ISO Standard: Is it worth it? Shammur Absar Chowdhury, Evgeny A. Stepanov, Giuseppe Riccardi Signals and Interactive Systems Lab Department of Information Engineering and Computer Science University of Trento, Italy {sachowdhury, stepanov, riccardi}@disi.unitn.it Abstract Spoken conversation corpora often adapt existing Dialogue Act (DA) annotation specifications, such as DAMSL, DIT++, etc., to task specific needs, yielding incompatible annotations; thus, limiting corpora re-usability. Recently accepted ISO standard for DA annotation – Dialogue Act Markup Language (DiAML) – is designed as domain and application independent. Moreover, the clear separation of dialogue dimensions and communicative functions, coupled with the hierarchical organization of the latter, allows for classification at different levels of granularity. However, re-annotating existing corpora with the new scheme might require significant effort. In this paper we test the utility of the ISO standard through comparative evaluation of the corpus-specific legacy and the semi-automatically transferred DiAML DA annotations on supervised dialogue act classification task. To test the domain independence of the resulting annotations, we perform cross-domain and data aggregation evaluation. Compared to the legacy annotation scheme, on the Italian LUNA Human-Human corpus, the DiAML annotation scheme exhibits better cross-domain and data aggregation classification performance, while maintaining comparable in-domain performance. Keywords: Dialogue Act, Dialogue corpora, Speech, DiAML, ISO Standard 1. Introduction Dialogue Acts (DA) are fundamental for the analysis of conversations: they carry communicative functions such as question, answer, expression of agreement and disagree- ment, etc. Consequently, the range of applications of DA analysis is quite wide and includes conversation summa- rization (both spoken and written), dialogue systems, etc.; and DAs have been extensively studied in both theoret- ical and computational linguistics. In the absence of a single commonly accepted standard, spoken corpora often adapt existing domain independent annotation schemes like DAMSL (Core and Allen, 1997), TRAINS (Traum, 1996), DIT++ (Bunt, 2005) to task-specific needs; thus, creating incompatible annotations. The supervised and unsuper- vised annotation and classification of DAs (e.g. (Joty et al., 2011)) and cross-domain and cross-media classification (e.g. forums, email, and spoken conversations (Joty et al., 2011; Tavafi et al., 2013)) have single important drawback: since the sets of considered DAs are not consistent, intro- duced cross-corpora mappings are at best generalizations or subsets. Recently accepted international ISO standard for DA anno- tation – Dialogue Act Markup Language (DiAML) (Bunt et al., 2010; Bunt et al., 2012) – could serve as a lingua franca for cross-corpora DA mapping. However, such mappings might require significant amount of manual re-annotation effort. The utility of abandoning the legacy annotations and manually or semi-automatically re-mapping them to the ISO standard could be tested under two conditions: (1) if the new annotation is equal or superior in supervised DA classification and (2) if it is indeed domain independent and allows both cross-domain application and pooling data from different domains (i.e., data aggregation). Thus, in this paper we presents experiments on the semi-automatic re-annotation of the Italian LUNA Corpus (Dinarelli et al., 2009) with DiAML and evaluation of the annotations in …

Content truncated to 500 words