An Investigation of Potential Function Designs for Neural CRF

Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2600–2609 November 16 - 20, 2020. c⃝2020 Association for Computational Linguistics 2600 An Investigation of Potential Function Designs for Neural CRF Zechuan Hu⋄, Yong Jiang†∗, Nguyen Bach†, Tao Wang†, Fei Huang†, Kewei Tu⋄∗ ⋄School of Information Science and Technology, ShanghaiTech University Shanghai Engineering Research Center of Intelligent Vision and Imaging Shanghai Institute of Microsystem and Information Technology, Chinese Academy of Sciences University of Chinese Academy of Sciences †DAMO Academy, Alibaba Group {huzch,tukw}@shanghaitech.edu.cn {yongjiang.jy,nguyen.bach,leeo.wangt,f.huang}@alibaba-inc.com Abstract The neural linear-chain CRF model is one of the most widely-used approach to sequence la- beling. In this paper, we investigate a series of increasingly expressive potential functions for neural CRF models, which not only inte- grate the emission and transition functions, but also explicitly take the representations of the contextual words as input. Our extensive ex- periments show that the decomposed quadri- linear potential function based on the vector representations of two neighboring labels and two neighboring words consistently achieves the best performance. 1 Introduction Sequence labeling is the task of labeling each token of a sequence. It is an important task in natural lan- guage processing and has a lot of applications such as Part-of-Speech Tagging (POS) (DeRose, 1988; Toutanova et al., 2003; Xin et al., 2018), Named Entity Recognition (NER) (Ritter et al., 2011; Ak- bik et al., 2019), Chunking (Tjong Kim Sang and Buchholz, 2000; Suzuki et al., 2006). The neural CRF model is one of the most widely- used approaches to sequence labeling and can achieve superior performance on many tasks (Col- lobert et al., 2011; Chen et al., 2015; Ling et al., 2015; Ma and Hovy, 2016; Lample et al., 2016a). It often employs an encoder such as a BiLSTM to compute the contextual vector representation of each word in the input sequence. The potential function at each position of the input sequence in a neural CRF is typically decomposed into an emis- sion function (of the current label and the vector representation of the current word) and a transition function (of the previous and current labels) (Liu et al., 2018; Yang et al., 2018). ∗Kewei Tu and Yong Jiang are the corresponding authors. … … Word Representations BiLSTM Encoder Softmax or various CRFs Inference Layer x1 x2 x3 xn h1 h2 h3 hn Figure 1: Neural architecture for sequence labeling In this paper, we design a series of increasingly expressive potential functions for neural CRF mod- els. First, we compute the transition function from label embeddings (Ma et al., 2016; Nam et al., 2016; Cui and Zhang, 2019) instead of label iden- tities. Second, we use a single potential function over the current word and the previous and current labels, instead of decomposing it into the emission and transition functions, leading to more expres- siveness. We also employ tensor decomposition in order to keep the potential function tractable. Thirdly, we take the representations of additional neighboring words as input to the potential func- tion, instead of solely relying on …

Content truncated to 500 words