ner and pos when nothing is capitalized

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 6256–6261, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 6256 ner and pos when nothing is capitalized Stephen Mayhew, Tatiana Tsygankova, Dan Roth University of Pennsylvania Philadelphia, PA, 19104 {mayhew, ttasya, danroth}@seas.upenn.edu Abstract For those languages which use it, capitaliza- tion is an important signal for the fundamen- tal NLP tasks of Named Entity Recognition (NER) and Part of Speech (POS) tagging. In fact, it is such a strong signal that model per- formance on these tasks drops sharply in com- mon lowercased scenarios, such as noisy web text or machine translation outputs. In this work, we perform a systematic analysis of so- lutions to this problem in English, modifying only the casing of the train or test data using lowercasing and truecasing methods. While prior work and first impressions might suggest training a caseless model, or using a truecaser at test time, we show that the most effective strategy is a concatenation of cased and lower- cased training data, producing a single model with high performance on both cased and un- cased text. As shown in our experiments, this result holds across tasks and input representa- tions. Finally, we show that our proposed solu- tion gives an 8% F1 improvement in mention detection on noisy out-of-domain Twitter data. 1 Introduction Many languages use capitalization in text, often to indicate named entities. For tasks that are con- cerned with named entities, such as named en- tity recognition (NER) and part of speech tagging (POS), this is an important signal, and models for these tasks nearly always retain it in training.1 But capitalization is not always available. For example, informal user-generated texts can have inconsistent capitalization, and similarly the out- puts of speech recognition or machine translation are traditionally without case. Ideally we would like a model to perform equally well on both cased and uncased text, in contrast with current models. 1For POS tagging, this happens in tagsets that explicitly mark proper nouns, such as the Penn Treebank tagset. Test Tool Task Cased Uncased BiLSTM-CRF w/ ELMo NER 92.45 34.46 BiLSTM-CRF w/ ELMo POS 97.85 88.66 Table 1: Modern tools trained on cased data perform well on cased test data, but poorly on uncased (low- ercased) test data. For NER, we evaluate on the testb set of CoNLL 2003, and the scores are reported as F1. For POS, we evaluate on PTB sections 22-24, and the scores represent accuracy. ELMo refers to contextual representations from Peters et al. (2018). Table 1 demonstrates how popular modern sys- tems trained on cased data perform well on cased data, but suffer dramatic performance drops when evaluated on lowercased text. Prior solutions have included models trained on lowercase text, or models that automatically re- cover capitalization from lowercase text, known as truecasing. There has a been a substantial body of literature on the effect of truecasing applied af- …

Content truncated to 500 words