Do LSTMs really work so well for PoS tagging? – A replication study

Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 727–736 Copenhagen, Denmark, September 7–11, 2017. c⃝2017 Association for Computational Linguistics Do LSTMs really work so well for PoS tagging? – A replication study Tobias Horsmann and Torsten Zesch Language Technology Lab Department of Computer Science and Applied Cognitive Science University of Duisburg-Essen, Germany {tobias.horsmann,torsten.zesch}@uni-due.de Abstract A recent study by Plank et al. (2016) found that LSTM-based PoS taggers consider- ably improve over the current state-of-the- art when evaluated on the corpora of the Universal Dependencies project that use a coarse-grained tagset. We replicate this study using a fresh collection of 27 cor- pora of 21 languages that are annotated with fine-grained tagsets of varying size. Our replication confirms the result in gen- eral, and we additionally find that the ad- vantage of LSTMs is even bigger for larger tagsets. However, we also find that for the very large tagsets of morphologically rich languages, hand-crafted morpholog- ical lexicons are still necessary to reach state-of-the-art performance. 1 Introduction Part-of-Speech (PoS) tagging is an important pro- cessing step for many NLP applications. When researchers want to use a PoS tagger, they would ideally choose an off-the-shelf PoS tagger which is optimized for a specific language. If a suited tagger is not available two options remain: a) im- plementation of your own tagger, which requires technical knowledge and experience, or b) us- ing an existing tagger and hope that the resulting model will be sufficiently accurate. One can as- sume that many taggers fit more languages than the one for which they have been constructed orig- inally. Ideally, researchers should be able to fall back to a well-evaluated language-independent tagger if no reference implementation for a lan- guage is available. A recent study by Plank et al. (2016) evalu- ated an LSTM PoS tagger and compared the re- sults to Conditional Random Fields (CRF) (Laf- ferty et al., 2001) and Hidden-Markov (HMM) implementations on corpora of various languages. Their evaluation concludes that the LSTM tagger reaches better results than the CRF and HMM tag- ger. The evaluation corpora were all annotated with a coarse-grained tagset with 17 tags. Thus, this LSTM tagger seems to be a well-performing, language-independent choice for learning models on coarse-grained tagsets. While for many tasks a coarse-grained tagset might be sufficient some tasks require more fine-grained tagsets. We, thus, consider it worthwhile to explore if the results are reproducible using corpora with fine-grained tagsets. We use the LSTM tagger pro- vided by Plank et al. (2016) and compare the re- sults likewise to CRF and an off-the-shelf HMM tagger implementation. We compile a fresh set of 27 corpora of 21 languages which uses the com- monly used fine-grained tagset of the respective language. We suggest these corpora as evaluation set for tasks which require fine-grained PoS tags, as all corpora are freely available for research pur- poses. Our intention is to replicate the findings of Plank et al. (2016), which have been achieved on a coarse-grained …

Content truncated to 500 words