An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 5569–5582, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 5569 An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks Lifu Tu1 ∗ Tianyu Liu2 ∗ Kevin Gimpel1 1 Toyota Technological Institute at Chicago, IL, USA 2 Institute of Computational Linguistics, Peking University, China {lifu, kgimpel}@ttic.edu, tianyu0421@pku.edu.cn Abstract Many tasks in natural language processing in- volve predicting structured outputs, e.g., se- quence labeling, semantic role labeling, pars- ing, and machine translation. Researchers are increasingly applying deep representation learning to these problems, but the structured component of these approaches is usually quite simplistic. In this work, we propose sev- eral high-order energy terms to capture com- plex dependencies among labels in sequence labeling, including several that consider the entire label sequence. We use neural pa- rameterizations for these energy terms, draw- ing from convolutional, recurrent, and self- attention networks. We use the framework of learning energy-based inference networks (Tu and Gimpel, 2018) for dealing with the dif- ficulties of training and inference with such models. We empirically demonstrate that this approach achieves substantial improvement us- ing a variety of high-order energy terms on four sequence labeling tasks, while having the same decoding speed as simple, local classi- fiers. We also find high-order energies to help in noisy data conditions.1 1 Introduction Conditional random fields (CRFs; Lafferty et al., 2001) have been shown to perform well in various sequence labeling tasks. Recent work uses rich neural network architectures to define the “unary” potentials, i.e., terms that only consider a single position’s label at a time (Collobert et al., 2011; Lample et al., 2016; Ma and Hovy, 2016; Strubell et al., 2018). However, “binary” potentials, which consider pairs of adjacent labels, are usually quite simple and may consist solely of a parameter or parameter vector for each unique label transition. ∗Equal contribution. 1Code is available at https://github.com/ tyliupku/Arbitrary-Order-Infnet Models with unary and binary potentials are gener- ally referred to as “first order” models. A major challenge with CRFs is the complexity of training and inference, which are quadratic in the number of output labels for first order models and grow exponentially when higher order depen- dencies are considered. This explains why the most common type of CRF used in practice is a first or- der model, also referred to as a “linear chain” CRF. One promising alternative to CRFs is structured prediction energy networks (SPENs; Belanger and McCallum, 2016), which use deep neural networks to parameterize arbitrary potential functions for structured prediction. While SPENs also pose chal- lenges for learning and inference, Tu and Gimpel (2018) proposed a way to train SPENs jointly with “inference networks”, neural networks trained to approximate structured arg max inference. In this paper, we leverage the frameworks of SPENs and inference networks to explore high- order energy functions for sequence labeling. Naively instantiating high-order energy terms can lead to a very large number of parameters to learn, so we instead develop concise neural parameteriza- tions …

Content truncated to 500 words