REPROLANG 2020: Automatic Proficiency Scoring of Czech, English, German, Italian, and Spanish Learner Essays

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 540–548 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 540 Estimating User Communication Styles for Spoken Dialogue Systems Juliana Miehle1, Isabel Feustel1, Julia Hornauer1, Wolfgang Minker1, Stefan Ultes2 1 Institute of Communications Engineering, Ulm University, Ulm, Germany {juliana.miehle, isabel.feustel, julia.hornauer, wolfgang.minker}@uni-ulm.de 2 Mercedes-Benz Research & Development, Sindlefingen, Germany stefan.ultes@daimler.com Abstract We present a neural network approach to estimate the communication style of spoken interaction, namely the stylistic variations elaborateness and directness, and investigate which type of input features to the estimator are necessary to achive good performance. First, we describe our annotated corpus of recordings in the health care domain and analyse the corpus statistics in terms of agreement, correlation and reliability of the ratings. We use this corpus to estimate the elaborateness and the directness of each utterance. We test different feature sets consisting of dialogue act features, grammatical features and linguistic features as input for our classifier and perform classification in two and three classes. Our classifiers use only features that can be automatically derived during an ongoing interaction in any spoken dialogue system without any prior annotation. Our results show that the elaborateness can be classified by only using the dialogue act and the amount of words contained in the corresponding utterance. The directness is a more difficult classification task and additional linguistic features in form of word embeddings improve the classification results. Afterwards, we run a comparison with a support vector machine and a recurrent neural network classifier. Keywords: Dialogue management, User adaptation, Supervised learning. 1. Introduction For humans, speech is the most natural form of interac- tion and it has been shown that people adapt their inter- action styles to one another across many levels of utter- ance production when communicating, e.g. by match- ing each other’s behaviour or synchronising the timing of behaviour (Burgoon et al., 2007; Niederhoffer and Pen- nebaker, 2002; Brennan, 1996; Pickering and Garrod, 2004; Nenkova et al., 2008). However, this adaptive be- haviour has rarely been addressed and implemented in a live spoken dialogue system. With the aim of designing such a spoken dialogue system which adapts to the user’s communication idiosyncrasies, we present a classification approach to automatically estimate the user’s communica- tion style during an ongoing dialogue. The estimated com- munication style can then be used in the dialogue manage- ment to adapt the system behaviour to the user, as depicted in Figure 1. To the best of our knowledge, this is the first work on automatic estimation of the user’s communication style in a dialogue system. We present the task of estimat- ing the user’s communication style and investigate the in- fluence of grammatical and linguistic features on its esti- mation. Our classifiers use only features that can be au- tomatically generated during an interaction with a spoken dialogue system (i.e. without any manual annotation). Even though intelligent assistants like Amazon Alexa, Ap- ple Siri, Google Assistant or Microsoft …

Content truncated to 500 words