Dissecting Span Identification Tasks with Performance Prediction

Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4881–4895, November 16–20, 2020. c⃝2020 Association for Computational Linguistics 4881 Dissecting Span Identification Tasks with Performance Prediction Sean Papay, Roman Klinger, and Sebastian Pad´o Institut f¨ur Maschinelle Sprachverarbeitung, University of Stuttgart Pfaffenwaldring 5b, 70569 Stuttgart, Germany {sean.papay,klinger,pado}@ims.uni-stuttgart.de Abstract Span identification (in short, span ID) tasks such as chunking, NER, or code-switching de- tection, ask models to identify and classify rel- evant spans in a text. Despite being a staple of NLP, and sharing a common structure, there is little insight on how these tasks’ properties in- fluence their difficulty, and thus little guidance on what model families work well on span ID tasks, and why. We analyze span ID tasks via performance prediction, estimating how well neural architectures do on different tasks. Our contributions are: (a) we identify key properties of span ID tasks that can inform performance prediction; (b) we carry out a large-scale experiment on English data, build- ing a model to predict performance for un- seen span ID tasks that can support architec- ture choices; (c), we investigate the parame- ters of the meta model, yielding new insights on how model and task properties interact to affect span ID performance. We find, e.g., that span frequency is especially important for LSTMs, and that CRFs help when spans are infrequent and boundaries non-distinctive. 1 Introduction Span identification is a family of analysis tasks that make up a substantial portion of applied NLP. Span identification (or short, span ID) tasks have in common that they identify and classify contiguous spans of tokens within a running text. Examples are named entity recognition (Nadeau and Sekine, 2007), chunking (Tjong Kim Sang and Buchholz, 2000), entity extraction (Etzioni et al., 2005), quo- tation detection (Pareti, 2016), keyphrase detection (Augenstein et al., 2017), or code switching (Prat- apa et al., 2018). In terms of complexity, span ID tasks form a middle ground between simpler anal- ysis tasks that predict labels for single linguistic units (such as lemmatization (Porter, 1980) or sen- timent polarity classification (Liu, 2012)) and more complex analysis tasks such as relation extraction, which combines span ID with relation identifica- tion (Zelenko et al., 2002; Adel et al., 2018). Due to the rapid development of deep learning, an abundance of model architectures is available for the implementation of span ID tasks. These include isolated token classification models (Berger et al., 1996; Chieu and Ng, 2003), probabilistic models such as hidden Markov models (Rabiner, 1989), maximum entropy Markov models (McCallum et al., 2000), and conditional random fields (Laf- ferty et al., 2001), recurrent neural networks such as LSTMs (Hochreiter and Schmidhuber, 1997), and transformers such as BERT (Devlin et al., 2019). Though we have some understanding what each of these models can and cannot learn, there is, to our knowledge, little work on systematically under- standing how different span ID tasks compare: are there model architectures that work well generally? Can we identify properties of span ID tasks that …

Content truncated to 500 words