Transforming Sequence Tagging Into A Seq2Seq Task

Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11856 - 11874 December 7-11, 2022 ©2022 Association for Computational Linguistics Transforming Sequence Tagging Into A Seq2Seq Task Karthik Raman∗, Iftekhar Naim, Jiecao Chen, Kazuma Hashimoto, Kiran Yalasangi, Krishna Srinivasan Google Research, Mountain View {karthikraman, inaim, chenjiecao, kazumah, kirancy, krishnaps}@google.com Abstract Pretrained, large, generative language models (LMs) have had great success in a wide range of sequence tagging and structured prediction tasks. Casting a sequence tagging task as a Seq2Seq one requires deciding the formats of the input and output sequences. However, we lack a principled understanding of the trade- offs associated with these formats (such as the effect on model accuracy, sequence length, multilingual generalization, hallucination). In this paper, we rigorously study different for- mats one could use for casting input text sen- tences and their output labels into the input and target (i.e., output) of a Seq2Seq model. Along the way, we introduce a new format, which we show to to be both simpler and more effective. Additionally the new format demonstrates significant gains in the multilin- gual settings – both zero-shot transfer learn- ing and joint training. Lastly, we find that the new format is more robust and almost com- pletely devoid of hallucination – an issue we find common in existing formats. With well over a 1000 experiments studying 14 different formats, over 7 diverse public benchmarks – including 3 multilingual datasets spanning 7 languages – we believe our findings provide a strong empirical basis in understanding how we should tackle sequence tagging tasks. 1 Introduction The advent of powerful generative language mod- els (LMs) such as T5 (Raffel et al., 2019) and mT5 (Xue et al., 2021) have unlocked new op- portunities. Recent works have shown great suc- cess applying these models to a wide range of sequence tagging and structured prediction tasks (Athiwaratkun et al., 2020; Du et al., 2021; Paolini et al., 2021). The key to leveraging these models is to cast the sequence tagging task as a suitable Seq2Seq task – by transforming the original input ∗ Corresponding Author text and output labels into the input and target (i.e., output) of a Seq2Seq model. Slot-filling example from Athiwaratkun et al. (2020) Input: Add Kent James to the Disney soundtrack Target: Add [ Kent James | artist ] to the [ Disney | playlist ] soundtrack SRL example from Paolini et al. (2021) (predicate = "sold") Input: The luxury auto maker last year [ sold ] 1,214 cars in the U.S. Target: [ The luxury auto maker |subject ] [ last year | temporal ] sold [ 1,214 cars | object ] [ in the U.S. | location ] Table 1: Examples of the TaggedSpans approach used in current works to transform NLP problems into the Seq2Seq setting. While the list of NLP problems tackled in this manner are diverse – Named Entity Recognition, Co-reference resolution, Semantic Role Labeling (SRL), Slot-filling, Entity and Relation extraction to name a few – …

Content truncated to 500 words