NAT: Noise-Aware Training for Robust Neural Sequence Labeling
38170Authors
nan
Abstract
AbstractSequence labeling systems should perform reliably not only under ideal conditions but also with corrupted inputs—as these systems often process user-generated text or follow an error-prone upstream component. To this end, we formulate the noisy sequence labeling problem, where the input may undergo an unknown noising process and propose two Noise-Aware Training (NAT) objectives that improve robustness of sequence labeling performed on perturbed input: Our data augmentation method trains a neural model using a mixture of clean and noisy samples, whereas our stability training algorithm encourages the model to create a noise-invariant latent representation. We employ a vanilla noise model at training time. For evaluation, we use both the original data and its variants perturbed with real OCR errors and misspellings. Extensive experiments on English and German named entity recognition benchmarks confirmed that NAT consistently improved robustness of popular sequence labeling models, preserving accuracy on the original input. We make our code and data publicly available for the research community.
Keywords
Full Content
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 133–138 July 5 - 10, 2020. c⃝2020 Association for Computational Linguistics 133 A Complete Shift-Reduce Chinese Discourse Parser with Robust Dynamic Oracle Shyh-Shiun Hung,1 Hen-Hsen Huang,2,3 and Hsin-Hsi Chen1,3 1 Department of Computer Science and Information Engineering, National Taiwan University, Taiwan 2 Department of Computer Science, National Chengchi University, Taiwan 3 MOST Joint Research Center for AI Technology and All Vista Healthcare, Taiwan shhung@nlg.csie.ntu.edu.tw, hhhuang@nccu.edu.tw, hhchen@ntu.edu.tw Abstract This work proposes a standalone, complete Chinese discourse parser for practical applica- tions. We approach Chinese discourse pars- ing from a variety of aspects and improve the shift-reduce parser not only by integrating the pre-trained text encoder, but also by em- ploying novel training strategies. We revise the dynamic-oracle procedure for training the shift-reduce parser, and apply unsupervised data augmentation to enhance rhetorical rela- tion recognition. Experimental results show that our Chinese discourse parser achieves the state-of-the-art performance. 1 Introduction Discourse parsing is one of the fundamental tasks in natural language processing (NLP). Typical types of discourse parsing include hierarchical dis- course parsing and shallow discourse parsing. The former is aimed at finding the relationships among a series of neighboring elementary discourse units (EDUs) and further building up a hierarchical tree structure (Mann and Thompson, 1988). Instead of establishing a tree structure, the latter finds the across-paragraph relations between all text units in a paragraph or a document. Based on Rhetorical Structure Theory Discourse Treebank (RST-DT) (Carlson et al., 2001a), hierarchical discourse pars- ing in English has been well-studied. This paper focuses on hierarchical discourse parsing in Chinese. Previous work on hierarchical Chinese discourse parsing is mostly based on the RST-style Chinese Discourse Treebank (Li et al., 2014). To distinguish from the other Chinese Dis- course Treebank (Zhou and Xue, 2012), which is annotated with the PDTB-style for shallow dis- course parsing, we use the term CDTB-14 to refer to the RST-style one and the term CDTB-12 to re- fer to the PDTB-style one. Kong and Zhou (2017) propose a pipeline framework and generate the dis- course parsing tree in a bottom-up way. Lin et al. (2018) propose an end-to-end system based on a recursive neural network (RvNN) to construct the parsing tree with a CKY-like algorithm. Sun and Kong (2018) use transition-based method with the stack augmented parser-interpreter neural network (SPINN) (Bowman et al., 2016) as the backbone model, helping their model make a better predic- tion with the previous information. In this work, we attempt to construct a com- plete Chinese discourse parser, which supports all the four sub-tasks in hierarchical discourse pars- ing, including EDU segmentation, tree structure construction, nuclearity labeling, and rhetorical re- lation recognition. Given a paragraph, our parser extracts all EDUs, builds the tree structure, iden- tifies the nucleuses, and recognizes the rhetorical relations of all internal nodes. We propose a re- vised dynamic-oracle procedure (Yu et al., 2018) for training the shift-reduce parser. Because of the limited training instances in CDTB-14, we also …
Content truncated to 500 words
Labeled Text
AbstractSequence labeling systems should perform reliably not only under ideal conditions but also with corrupted inputs—as these systems often process user-generated text or follow an error-prone upstream component. To this end, we formulate the noisy sequence labeling problem, where the input may undergo an unknown noising process and propose two Noise-Aware Training (NAT) objectives that improve robustness of sequence labeling performed on perturbed input: Our data augmentation method trains a neural model using a mixture of clean and noisy samples, whereas our stability training algorithm encourages the model to create a noise-invariant latent representation. We employ a vanilla noise model at training time. For evaluation, we use both the original data and its variants perturbed with real OCR errors and misspellings. Extensive experiments on English and German named entity recognition benchmarks confirmed that NAT consistently improved robustness of popular sequence labeling models, preserving accuracy on the original input. We make our code and data publicly available for the research community.
Publication Details
- Paper ID:
- 38170
- Year:
- 2020
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan