Sequence to Sequence Convolutional Neural Network for Automatic Spelling Correction

Sequence to Sequence Convolutional Neural Network for Automatic Spelling Correction Daniel Hládek, Matúš Pleva, Ján Staš, Department of Electronics and Multimedia Communications Technical University of Košice, Slovakia daniel.hladek@tuke.sk,matus.pleva@tuke.sk,jan.stas@tuke.sk Yuan-Fu Liao Department of Electronic Engineering National Taipei University of Technology yfliao@mail.ntut.edu.tw Abstract The paper proposes a system that compensates most of the noise in a text in natural language caused by technical imperfection of the input device such as keyboard or scanner with optical character recognition, quick typing, or writer incompetence. Correcting the spelling errors in the text improves the performance of the following natural language processing. The incorrect sequence of characters is transcribed into another sequence of correct characters by a neural network with encoder-decoder architecture. Our approach to automatic spelling correction considers characters in an erroneous sentence as words of the source languages. The neural network searches for the best sequence of output characters for the given input. The proposed approach for spelling correction does not require any or minimal amount of training data. Instead, the error model is expressed by a simple component that distorts unannotated data and creates any necessary quantity of training examples for a neural network. The experimental results show that the presented approach significantly improves the distorted data (from 50% WER to 0.09% WER) with distortion lower than 1.5% WER. Keywords: automatic spelling correction, sequence to sequence, encoder-decoder, deep learning. The 2019 Conference on Computational Linguistics and Speech Processing ROCLING 2019, pp. 102-111 ©The Association for Computational Linguistics and Chinese Language Processing 102 1. Introduction Written or scanned text is often not in the intended form. Writer or the input device often generate deviations that make it less understandable. The errors are usually not a problem in casual communication but make machine processing more complicated. Removal of spelling errors helps with the following processing of the text in natural language. Automatic spelling correction (ASC) is an essential part of the processing of the documents with noisy data in natural language. ASC helps to recover the intended canonical form of the text and improves the quality of the input data for the following natural language processing (NLP) components. It supports processing of digitized documents, automated proofreaders, or information retrieval systems (e.g TREC-5 confusion track [1]). The main motivation for this work is an improvement of the training data for language model for speech recognition [2]. 2. State of the art The task of automatic error-correction is to generate the most likely correct word-for ms given a misspelled word-form [3]. Previous approaches to ASC, such as correcting spelling errors in the Chinese language [4] use classical statistical methods, such as the hidden Markov model, n-gram language model, log- linear regression, or forward-backward algorithm [5]. The usual form of mathematical formalism is a noisy channel proposed by Shannon [6]. Shannon defines the ASC of a possibly incorrect word 𝑠 as finding the best correction candidate 𝑤𝑏 from a list of possible correction candidates 𝑤𝑖 ∈ 𝑊 (𝑊 is a valid word dictionary) with the best unnormalized probability …

Content truncated to 500 words