A CUP of CoFee: A large Collection of feedback Utterances Provided with communicative function annotations

A CUP of COFEE: A Large Collection of Feedback Utterances Provided with Communicative Function Annotations Laurent Pr´evot, Jan Gorisch, Roxane Bertrand LPL: Aix-Marseille Universit´e, Institut f¨ur Deutsche Sprache, LPL: Aix-Marseille Universit´e Aix-en-Provence (France), Mannheim (Germany), Aix-en-Provence (France) laurent.prevot@lpl-aix.fr, gorisch@ids-mannheim.de, roxane.bertrand@lpl-aix.fr Abstract There have been several attempts to annotate communicative functions to utterances of verbal feedback in English previously. Here, we suggest an annotation scheme for verbal and non-verbal feedback utterances in French including the categories base, attitude, previous and visual. The data comprises conversations, maptasks and negotiations from which we extracted ca. 13,000 candidate feedback utterances and gestures. 12 students were recruited for the annotation campaign of ca. 9,500 instances. Each instance was annotated by between 2 and 7 raters. The evaluation of the annotation agreement resulted in an average best-pair kappa of 0.6. While the base category with the values acknowledgement, evaluation, answer, elicit and other achieves good agreement, this is not the case for the other main categories. The data sets, which also include automatic extractions of lexical, positional and acoustic features, are freely available and will further be used for machine learning classification experiments to analyse the form-function relationship of feedback. Keywords: Conversational Feedback, Communicative Functions, Annotator Agreement 1. Introduction Feedback utterances are among the most frequent in dia- logue. Feedback is also a crucial aspect of linguistic theo- ries taking interaction into account. For example it is of- ten claimed that linguistic resources, such as the use of prosody, is used by interactional participants for specific communicative ends. In order to systematically test this relationship between communicative functions and the lin- guistic form of feedback, two tasks are required. One task is the extraction of linguistic features, including prosodic, lexical, positional, etc. features. The other task involves the annotation of communicative functions related to feedback utterances. In this paper we address the annotation task. A range of communicative functions have been identified in previous research, however the annotation of feedback communicative functions is still a notoriously difficult task, as the inter-annotator agreement measures show (if the an- notations are evaluated at all in such papers). The annota- tion involves an interpretative process that integrates vari- ous sources of information, such as auditory/acoustic and visual information. The study reported in this paper takes place in the project COFEE1 (Pr´evot and Bertrand, 2012) that aims to use, among other methodologies, quantitative clues to decipher the form-function relationship within feedback utterances. More precisely, we are interested in the creation of (large) datasets composed of feedback utterances annotated with communicative functions. From these datasets, we conduce quantitative (statistical) linguistics tests as well as machine learning classification experiments. We analyzed material from three corpora where partici- pants (i) have a free conversation, (ii) do a Map Task, (iii) do a conversation that also involves some negotiation. First, we extracted all potential feedback: the verbal feedback utterances via the orthographic transcription, and the non- verbal feedback via a visual pre-segmentation of the videos. 1http://cofee.hypotheses.org/ Second, we performed an annotation campaign with 12 …

Content truncated to 500 words