TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialog

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 267–280 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 267 A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction Masato Mita1,2 Shun Kiyono1,2 Masahiro Kaneko3,1 Jun Suzuki2,1 Kentaro Inui2,1 1RIKEN Center for Advanced Intelligence Project 2Tohoku University 3Tokyo Metropolitan University {masato.mita, shun.kiyono}@riken.jp kaneko-masahiro@ed.tmu.ac.jp {jun.suzuki, inui}@ecei.tohoku.ac.jp Abstract Existing approaches for grammatical error cor- rection (GEC) largely rely on supervised learn- ing with manually created GEC datasets. How- ever, there has been little focus on verifying and ensuring the quality of the datasets, and on how lower-quality data might affect GEC performance. We indeed found that there is a non-negligible amount of “noise” where errors were inappropriately edited or left uncorrected. To address this, we designed a self-refinement method where the key idea is to denoise these datasets by leveraging the prediction consis- tency of existing models, and outperformed strong denoising baseline methods. We further applied task-specific techniques and achieved state-of-the-art performance on the CoNLL- 2014, JFLEG, and BEA-2019 benchmarks. We then analyzed the effect of the proposed de- noising method, and found that our approach leads to improved coverage of corrections and facilitated fluency edits which are reflected in higher recall and overall performance. 1 Introduction Grammatical error correction (GEC) is often con- sidered a variant of machine translation (MT) (Brockett et al., 2006; Junczys-Dowmunt et al., 2018) due to their structural similarity–“translating” from source ungrammatical text to target gram- matical text. At present, several neural encoder- decoder (EncDec) approaches have been intro- duced for this task and have achieved remarkable results (Chollampatt and Ng, 2018; Zhao et al., 2019; Kiyono et al., 2019). EncDec models tend to further improve in performance with increasing data size (Koehn and Knowles, 2017; Sennrich and Zhang, 2019), however, this is not necessarily true in GEC. For example, Lo et al. (2018) reported that an EncDec-based GEC model trained on EF- 1 : Errors are inappropriately edited Source: I want to discuss about the education. Target: I want to discuss of the education. 2 : Errors are left uncorrected Source: We discuss about our sales target. Target: We discuss about our sales target. Table 1: Example of an inappropriately corrected error and an unchanged error in EFCamDat. We consider these types of errors to be dataset noise that might hin- der GEC model performance. CamDat (Geertzen et al., 2013)1, the largest pub- licly available learner corpus as of today (two mil- lion sentence pairs), was outperformed by a model trained on a smaller dataset (e.g., 720K pairs). They hypothesized that this may be due to the noisiness of EFCamDat, i.e., the presence of sentence pairs whose correction still contained grammatical errors due to inappropriate edits or being left uncorrected. For example, in Table 1, “discuss about” should most likely have been corrected to “discuss”, and “are discussing”, respectively. We confirmed that there is a non-negligible amount of noise in com- monly used GEC datasets (Section 3). We recognise data noise as a …

Content truncated to 500 words