A Study of Incorrect Paraphrases in Crowdsourced User Utterances

Proceedings of NAACL-HLT 2019, pages 295–306 Minneapolis, Minnesota, June 2 - June 7, 2019. c⃝2019 Association for Computational Linguistics 295 A Study of Incorrect Paraphrases in Crowdsourced User Utterances Mohammad-Ali Yaghoub-Zadeh-Fard, Boualem Benatallah, Moshe Chai Barukh and Shayan Zamanirad University of New South Wales, Sydney {m.yaghoubzadehfard,b.benatallah,mosheb,shayanz}@unsw.edu.au Abstract Developing bots demands high quality train- ing samples, typically in the form of user ut- terances and their associated intents. Given the fuzzy nature of human language, such datasets ideally must cover all possible utterances of each single intent. Crowdsourcing has widely been used to collect such inclusive datasets by paraphrasing an initial utterance. However, the quality of this approach often suffers from var- ious issues, particularly language errors pro- duced by unqualified crowd workers. More so, since workers are tasked to write open-ended text, it is very challenging to automatically asses the quality of paraphrased utterances. In this paper, we investigate common crowd- sourced paraphrasing issues, and propose an annotated dataset called Para-Quality, for de- tecting the quality issues. We also investigate existing tools and services to provide baselines for detecting each category of issues. In all, this work presents a data-driven view of incor- rect paraphrases during the bot development process, and we pave the way towards auto- matic detection of unqualified paraphrases. 1 Introduction With the increasing advances in deep learning as well as natural language processing, a new gen- eration of conversational agents is attracting sig- nificant attention (Dale, 2016). Also known as dialogue systems, virtual assistants, chatbots or simply bots (Campagna et al., 2017; Su et al., 2017), some advanced bots are now designed to perform complex tasks (e.g., flight booking), many of which are built using machine learning tech- niques. At the heart of building such task-oriented bots lies the challenge of accurately capturing the user’s intent (e.g., find cafes in Chicago), and then extracting its entities to service the request (e.g term= “cafes”, location=“Chicago”). However, its success relies heavily on obtaining both, large and high quality corpora of training samples show- ing mappings between sample utterances and in- tents. This is necessary given the ambiguous na- ture of the human language (Wasow et al., 2005) and large variations of expressions (Wang et al., 2012; Zamanirad et al., 2017). A lack of variations in training samples can re- sult in incorrect intent detection and consequently execution of undesirable tasks (e.g., booking an expensive hotel instead of a cheap room) (Hen- derson et al., 2018). Likewise, quality issues in the training samples can lead to unmitigated dis- asters (Neff and Nagy, 2016) as it happened to Microsoft’s Tay by making a huge number of of- fensive commentaries due to biases in the train- ing data (Henderson et al., 2018). It is there- fore not surprising that research and development into training data acquisition for bots has received significant consideration (Campagna et al., 2017; Kang et al., 2018). Collecting training samples usually involves two primary steps: (i) firstly, obtaining an ini- tial utterance for a given user intent (e.g., …

Content truncated to 500 words