Towards an Automatic Assessment of Crowdsourced Data for NLU

Towards an Automatic Assessment of Crowdsourced Data for NLU Patricia Braunger, Wolfgang Maier, Jan Wessling, Maria Schmidt Daimler AG Sindelfingen, Germany {patricia.braunger, wolfgang.mw.maier, jan.wessling, maria.m.schmidt}@daimler.com Abstract Recent development of spoken dialog systems has moved away from a command-style input and aims at allowing a natural input style. Obtaining suitable data for training and testing such systems is a significant challenge. We investigate with which methods data elicited via crowdsourcing can be assessed with respect to its naturalness and usefulness. Since the criteria with which to assess usefulness depend on the application purpose of crowdsourced data we investigate various facets such as noisy data, naturalness and building natural language understanding (NLU) models. Our results show that valid data can be automatically identified with the help of a word based language model. A comparison of crowdsourced data and system usage data on lexical, syntactic and pragmatic level reveals detailed information on the differences between both data sets. However, we show that using crowdsourced data for training NLU services achieves similar results as system usage data. Keywords: crowdsourcing, spoken dialog system, natural language understanding, training data 1. Introduction A major goal of current spoken dialog systems (SDS) de- velopment is to obtain a more natural and human-like style of communication. Such a communication style presup- poses an understanding of all utterances which are associ- ated with a specific semantic meaning. Thanks to the ad- vent of statistical data-driven approaches, recent systems have been able to interpret freely spoken user input. How- ever, obtaining suitable data for their training and evalua- tion is a significant challenge. In order to increase the amount of data for training and evaluation of natural language understanding (NLU), re- searchers are turning to crowdsourcing instead of collecting data from Wizard of Oz experiments or using handcrafted grammars (see, e.g., Callison-Burch and Dredze (2010)). While the benefits of crowdsourcing can include expedi- tion, individuality of people, and low costs (see, e.g., Eske- nazi et al. (2013)), it is often criticized for poor standards as it is difficult to control the quality of work when requesting complex tasks (Eskenazi et al., 2013). Since the commu- nity of crowd workers is acknowledged for being Internet savvy young adults, chiefly between ages 18 and 35, the findings are not representative for any target group. In addi- tion, crowdsourced data intended to be used for improving SDS do not necessarily reflect real system usage. Therefore we investigate to which extent crowdsourced data are suit- able for the development of human-like SDS. Specifically, we want to provide answers to the following questions: • Do data elicited via crowdsourcing reflect real system usage and naturalness? • Are data elicited via crowdsourcing suitable to encour- age the development of high-quality NLU modules? Thereby, we aim to propose methods with which to assess crowdsourced data for NLU. In order to establish a baseline of what requirements the utterances collected via crowd- sourcing must fulfill, we conduct a study in which free user utterances are collected for an actual …

Content truncated to 500 words