Improving Named Entity Recognition with Attentive Ensemble of Syntactic Information

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 407–414 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 407 Context-aware Stand-alone Neural Spelling Correction Xiangci Li ∗ University of Texas at Dallas Richardson, TX lixiangci8@gmail.com Hairong Liu Baidu USA Sunnyvale, CA liuhairong@baidu.com Liang Huang Baidu USA Sunnyvale, CA lianghuang@baidu.com Abstract Existing natural language processing systems are vulnerable to noisy inputs resulting from misspellings. On the contrary, humans can easily infer the corresponding correct words from their misspellings and surrounding con- text. Inspired by this, we address the stand- alone spelling correction problem, which only corrects the spelling of each token without ad- ditional token insertion or deletion, by utiliz- ing both spelling information and global con- text representations. We present a simple yet powerful solution that jointly detects and cor- rects misspellings as a sequence labeling task by fine-turning a pre-trained language model. Our solution outperform the previous state-of- the-art result by 12.8% absolute F0.5 score. 1 Introduction A spelling corrector is an important and ubiquitous pre-processing tool in a wide range of applications, such as word processors, search engines and ma- chine translation systems. Having a surprisingly robust language processing system to denoise the scrambled spellings, humans can relatively easily solve spelling correction (Rawlinson, 1976). How- ever, spelling correction is a challenging task for a machine, because words can be misspelled in var- ious ways, and a machine has difficulties in fully utilizing the contextual information. Misspellings can be categorized into non-word, which is out-of-vocabulary, and the opposite, real- word misspellings (Klabunde, 2002). The dictio- nary look-up method can detect non-word mis- spellings, while real-word spelling errors are harder to detect, since these misspellings are in the vocab- ulary (Mays et al., 1991; Wilcox-O’Hearn et al., 2008). In this work, we address the stand-alone (Li et al., 2018) spelling correction problem. It only ∗Work performed during internship with Baidu USA Word+Char Encoder Model i love you affine i <unk> you i l y CLS CLS CLS a o v u e 𝐸𝑛𝑐𝑜𝑑𝑒𝑟!"#$ 𝐸𝑛𝑐𝑜𝑑𝑒𝑟%&'# i lave you i love you Subword Encoder Model 𝐸𝑛𝑐𝑜𝑑𝑒𝑟()* B-i I-love B-you B-love affine i ##ve you la i love you i lave you i la ##ve you 𝑆!"# ∗ Figure 1: A schematic illustration of our approach. Left: combined word-level and character-level encoder model. Right: subword-level model using BIO2 tag- ging scheme (Sang and Veenstra, 1999). corrects the spelling of each token without intro- ducing new tokens or deleting tokens, so that the original information is maximally preserved for the down-stream tasks. We formulate the stand-alone spelling correc- tion as a sequence labeling task and jointly detect and correct misspellings. Inspired by the human language processing system, we propose a novel solution on the following aspects: (1) We encode both spelling information and global context infor- mation in the neural network. (2) We enhance the real-word correction performance by initializing the model from a pre-trained language model (LM). (3) We strengthen the model’s robustness on unseen non-word misspellings by augmenting the …

Content truncated to 500 words