Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction

Proceedings of NAACL-HLT 2018, pages 619–628 New Orleans, Louisiana, June 1 - 6, 2018. c⃝2018 Association for Computational Linguistics Noising and Denoising Natural Language: Diverse Backtranslation for Grammar Correction Ziang Xie, Guillaume Genthial, Stanley Xie, Andrew Y. Ng, Dan Jurafsky Computer Science Department, Stanford University {zxie,genthial,stanxie,ang}@cs.stanford.edu,jurafsky@stanford.edu Abstract Translation-based methods for grammar cor- rection that directly map noisy, ungrammat- ical text to their clean counterparts are able to correct a broad range of errors; how- ever, such techniques are bottlenecked by the need for a large parallel corpus of noisy and clean sentence pairs. In this paper, we con- sider synthesizing parallel data by noising a clean monolingual corpus. While most pre- vious approaches introduce perturbations us- ing features computed from local context win- dows, we instead develop error generation processes using a neural sequence transduc- tion model trained to translate clean exam- ples to their noisy counterparts. Given a corpus of clean examples, we propose beam search noising procedures to synthesize ad- ditional noisy examples that human evalua- tors were nearly unable to discriminate from nonsynthesized examples. Surprisingly, when trained on additional data synthesized us- ing our best-performing noising scheme, our model approaches the same performance as when trained on additional nonsynthesized data. 1 Introduction Correcting noisy, ungrammatical text remains a challenging task in natural language processing. Ideally, given some piece of writing, an error cor- rection system would be able to fix minor typo- graphical errors, as well as grammatical errors that involve longer dependencies such as nonidiomatic phrasing or errors in subject-verb agreement. Ex- isting methods, however, are often only able to correct highly local errors, such as spelling errors or errors involving articles or prepositions. Classifier-based approaches to error correction are limited in their ability to capture a broad range of error types (Ng et al., 2014). Machine translation-based approaches—that instead trans- Noise (Decode) Denoise "New Orleans" "NLP" "new Orleens" "nlp" 2. Synthesize data "new Orleens" "nlp" "New Orleans" "NLP" 3. Train/decode denoising model 1. Train noising model Noise (Train) "Louisiana" "conference" "louiseana" "conferance" Figure 1: Overview of method. We first train a noise model on a seed corpus, then apply noise during de- coding to synthesize data that is in turn used to train the denoising model. late noisy, ungrammatical sentences to clean, cor- rected sentences—can flexibly handle a large vari- ety of errors; however, such approaches are bottle- necked by the need for a large dataset of source- target sentence pairs. To address this data sparsity problem, we pro- pose methods for synthesizing noisy sentences from clean sentences, thus generating an addi- tional artificial dataset of noisy and clean sentence pairs. A simple approach to noise clean text is to noise individual tokens or bigrams, for example by replacing each token with a random draw from the unigram distribution. This type of approach, how- ever, tends to generate highly unrealistic noise and fails to capture phrase-level phenomena. Other rule-based approaches fail to capture a diverse set of error types. We consider a method inspired by …

Content truncated to 500 words