A Semi-Markov Structured Support Vector Machine Model for High-Precision Named Entity Recognition

Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5862–5866 Florence, Italy, July 28 - August 2, 2019. c⃝2019 Association for Computational Linguistics 5862 A Semi-Markov Structured Support Vector Machine Model for High-Precision Named Entity Recognition Ravneet Arora, Chen-Tse Tsai, Ketevan Tsereteli, Prabhanjan Kambadur, Yi Yang∗ Bloomberg L.P., *ASAPP Inc. {rarora62,ctsai54,ktsereteli1,pkambadur}@bloomberg.net, *yyang@asapp.com Abstract Named entity recognition (NER) is the back- bone of many NLP solutions. F1 score, the harmonic mean of precision and recall, is often used to select/evaluate the best models. How- ever, when precision needs to be prioritized over recall, a state-of-the-art model might not be the best choice. There is little in the litera- ture that directly addresses training-time mod- ifications to achieve higher precision informa- tion extraction. In this paper, we propose a neural semi-Markov structured support vec- tor machine model that controls the precision- recall trade-off by assigning weights to differ- ent types of errors in the loss-augmented infer- ence during training. The semi-Markov prop- erty provides more accurate phrase-level pre- dictions, thereby improving performance. We empirically demonstrate the advantage of our model when high precision is required by com- paring against strong baselines based on CRF. In our experiments with the CoNLL 2003 dataset, our model achieves a better precision- recall trade-off at various precision levels. 1 Introduction Named Entity Recognition (NER) is the task of locating and categorizing phrases into a closed set of classes, such as organizations, people, and loca- tions. NER is an information extraction task that is important for understanding large bodies of text and is an essential component for many natural language processing (NLP) pipelines. The most common evaluation metric for information extrac- tion tasks is F1, which is the harmonic mean be- tween precision and recall: that is, false positives and false negatives are weighted equally. In certain real-world applications (e.g., medicine and finance), extracting wrong informa- tion is much worse than extracting nothing: hence, ∗Work conducted while working at Bloomberg L.P. in such domains, high precision is emphasized. Trade-offs between precision and recall have been well researched for classification (Joachims, 2005; Jansche, 2005; Cortes and Mohri, 2004). However, barring studies on inference-time heuristics, there is limited work on training precision-oriented sequence tagging models. In this paper, we present a method for training precision-driven NER models. By defining custom loss objectives for the structured SVM (SSVM) model, we extend cost- sensitive learning (Domingos, 1999; Margineantu, 2001) to sequence tagging problems. A difficulty in applying cost-sensitive learning to NER is that the model needs to operate on segmentations of the input sentence and the labels of the segments. Inspired by semi-Markov CRF (Sarawagi and Co- hen, 2005), we propose a semi-Markov SSVM model that scores and labels consecutive tokens together, which allows us to directly interact with the segment-level errors in the precision-beneficial loss of the SSVM model. We compare our semi-Markov SSVM model with several competitive inference-time baselines that have been proposed for high-precision NER. Our results show that our model outperforms com- petitive …

Content truncated to 500 words