Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsing

Unleashing the True Potential of Sequence-to-Sequence Models for Sequence Tagging and Structure Parsing Han He Department of Computer Science Emory University Atlanta, GA 30322 USA han.he@emory.edu Jinho D. Choi Department of Computer Science Emory University Atlanta, GA 30322 USA jinho.choi@emory.edu Abstract Sequence-to-Sequence (S2S) models have achieved remarkable success on various text generation tasks. However, learning complex structures with S2S models remains challeng- ing as external neural modules and additional lexicons are often supplemented to predict non-textual outputs. We present a systematic study of S2S modeling using contained de- coding on four core tasks: part-of-speech tag- ging, named entity recognition, constituency, and dependency parsing, to develop efficient exploitation methods costing zero extra param- eters. In particular, 3 lexically diverse lineari- zation schemas and corresponding constrained decoding methods are designed and evaluated. Experiments show that although more lexical- ized schemas yield longer output sequences that require heavier training, their sequences being closer to natural language makes them easier to learn. Moreover, S2S models using our constrained decoding outperform other S2S approaches using external resources. Our best models perform better than or comparably to the state-of-the-art for all 4 tasks, lighting a promise for S2S models to generate non- sequential structures. 1 Introduction Sequence-to-Sequence (S2S) models pretrained for language modeling (PLM) and denoising ob- jectives have been successful on a wide range of NLP tasks where both inputs and outputs are se- quences (Radford et al., 2019; Raffel et al., 2020; Lewis et al., 2020; Brown et al., 2020). However, for non-sequential outputs like trees and graphs, a procedure called linearization is often required to flatten them into ordinary sequences (Li et al., 2018; Fern´andez-Gonz´alez and G´omez-Rodr´ıguez, 2020; Yan et al., 2021; Bevilacqua et al., 2021; He and Choi, 2021a), where labels in non-sequential structures are mapped heuristically as individual tokens in sequences, and numerical properties like indices are either predicted using an external de- coder such as Pointer Networks (Vinyals et al., 2015a) or cast to additional tokens in the vocab- ulary. While these methods are found to be effec- tive, we hypothesize that S2S models can learn complex structures without adapting such patches. To challenge the limit of S2S modeling, BART (Lewis et al., 2020) is finetuned on four tasks with- out extra decoders: part-of-speech tagging (POS), named entity recognition (NER), constituency pars- ing (CON), and dependency parsing (DEP). Three novel linearization schemas are introduced for each task: label sequence (LS), label with text (LT), and prompt (PT). LS to PT feature an increasing number of lexicons and a decreasing number of labels, which are not in the vocabulary (Section 3). Every schema is equipped with a constrained decoding algorithm searching over valid sequences (Section 4). Our experiments on three popular datasets de- pict that S2S models can learn these linguistic structures without external resources such as in- dex tokens or Pointer Networks. Our best models perform on par with or better than the other state- of-the-art models for all four tasks (Section 5). Finally, a detailed analysis is provided to …

Content truncated to 500 words