On the Machine Learning of Ethical Judgments from Natural Language

Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 76 - 84 July 10-15, 2022 ©2022 Association for Computational Linguistics ErAConD: Error Annotated Conversational Dialog Dataset for Grammatical Error Correction Xun Yuan∗2, Derek Pham∗1, Sam Davidson3 and Zhou Yu1 1Columbia University, 2Zhejiang University, 3University of California, Davis {dp3081, zy2461}@columbia.edu yuanxun@zju.edu.cn, ssdavidson@ucdavis.edu Abstract Currently available grammatical error correc- tion (GEC) datasets are compiled using essays or other long-form text written by language learners, limiting the applicability of these datasets to other domains such as informal writing and conversational dialog. In this pa- per, we present a novel GEC dataset consisting of parallel original and corrected utterances drawn from open-domain chatbot conversa- tions; this dataset is, to our knowledge, the first GEC dataset targeted to a human-machine conversational setting. We also present a de- tailed annotation scheme which ranks errors by perceived impact on comprehension, mak- ing our dataset more representative of real- world language learning applications. To demonstrate the utility of the dataset, we use our annotated data to fine-tune a state-of-the- art GEC model. Experimental results show the effectiveness of our data in improving GEC model performance in a conversational sce- nario. 1 Introduction In recent years, both researchers and businesses have attempted to build effective educational chat- bots to help language learners improve their con- versational skills in a second language (primarily English) (Huang et al., 2021). However, many such systems, such as GenieTutor Plus (Huang et al., 2017), use rule-based dialog engines, and thus do not take advantage of recent developments in dialog generation using Transformer models, which have vastly improved the quality of mod- ern chatbots (Liang et al., 2020). Extant dialog systems for conversational language learning can be broadly classified into two types. In the first type, the chatbot serves as a teacher and repeatedly asks the user questions to test acquisition of spe- cific words, syntax, and other pedagogical targets. ∗Authors contributed equally to this work. In the second type, the chatbot serves as a conver- sational partner, encouraging users to chat with it and, in some cases, providing corrective feedback to learners (Fryer et al., 2020). It is this latter type we hope to improve using our proposed dataset. Grammatical error correction (GEC) models are needed to generate appropriate corrective feed- back for this second type of educational chatbot. However, nearly all current GEC datasets focus on written essays, a domain which differs markedly from conversational speech in both syntax and style. As a result, datasets drawn from writ- ten sources, such as student essays, produce poor results when applied to dialog (Davidson et al., 2019). There currently exists one dataset of error- annotated conversational utterances by English second language learners on which researchers can train and evaluate conversational GEC mod- els, the Teacher-Student Chatroom Corpus (Caines et al., 2020); this data is generated in the context of human-human interaction, specifically interac- tions between a teacher and a second language learner student. However, …

Content truncated to 500 words