Structured Refinement for Sequential Labeling

Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1873–1884 August 1–6, 2021. ©2021 Association for Computational Linguistics 1873 Structured Refinement for Sequential Labeling Yiran Wang1∗, Hiroyuki Shindo2, Yuji Matsumoto3, Taro Watanabe2 1National Institute of Information and Communications Technology (NICT), Kyoto, Japan 2Nara Institute of Science and Technology (NAIST), Nara, Japan 3RIKEN Center for Advanced Intelligence Project (AIP), Tokyo, Japan yiran.wang@nict.go.jp, shindo@is.naist.jp, yuji.matsumoto@riken.jp, taro@is.naist.jp Abstract Filtering target-irrelevant information through hierarchically refining hidden states has been demonstrated to be effective for obtaining in- formative representations. However, previous work simply relies on locally normalized at- tention without considering possible labels at other time steps, the capacity for modeling long-term dependency relations is thus limited. In this paper, we propose to extend previous work with globally normalized attention, e.g., structured attention, to leverage structural in- formation for more effective representation re- finement. We also propose two implementa- tion tricks to accelerate CRF computation and an initialization trick for Chinese character em- beddings to further improve performance. We provide extensive experimental results on vari- ous datasets to show the effectiveness and effi- ciency of our proposed method. 1 Introduction Sequential labeling tasks, e.g., named entity recog- nition (NER) and part-of-speech (POS) tagging, play an important role in natural language process- ing. Figure 1 shows two examples of sequential labeling tasks. Early studies focused on introducing rich features to improve performance. For exam- ple, to handle out-of-vocabulary words by introduc- ing morphological features, Lample et al. (2016) and Ma and Hovy (2016) leveraged character-level features, whereas Heinzerling and Strube (2019) exploited subword-level features. Moreover, intro- ducing long-term dependency features is also found to be beneficial for sequential labeling. Jie and Lu (2019) attempted to explicitly exploit dependency relations with additional annotations, while Zhang et al. (2018) and Chen et al. (2019) endeavored to learn these relations implicitly with more complex encoders. ∗This work was done when the first author was at NAIST. The DT financing NN system NN is VBD created VBN in IN the DT new JJ law NN The O chairman O of O the B-ORG Senate I-ORG Finance I-ORG Committee E-ORG Figure 1: Examples of NER (top) and POS tagging (bottom). For NER, “the Senate Finance Committee” is a named entity of type ORG (organization). The prefixes S-, I-, or E- indicate this word is located at the beginning, intermediate, or ending of the current named entity, while O signifies this word is outside any named entity. In the case of POS tagging, each tag is a part-of-speech category. For instance, NN represents a singular noun and VBN is the past participle of a verb. However, as Tishby and Zaslavsky (2015) pointed out, features are not created equal, only the target-relevant features are profitable for improv- ing model performance. Recently, Cui and Zhang (2019) proposed a hierarchically-refined label at- tention network (LAN), which explicitly leverages label embeddings and captures long-term label de- pendency relations through multiple refinements layers. Individually picking up the most likely label at each time step is undoubtedly …

Content truncated to 500 words