Non-lexical neural architecture for fine-grained POS Tagging

Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 232–237, Lisbon, Portugal, 17-21 September 2015. c⃝2015 Association for Computational Linguistics. Non-lexical neural architecture for fine-grained POS Tagging Matthieu Labeau, Kevin L¨oser, Alexandre Allauzen Universit´e Paris-Sud and LIMSI-CNRS, Rue John von Neumann 91403 Orsay cedex France firstname.lastname@limsi.fr Abstract In this paper we explore a POS tagging ap- plication of neural architectures that can infer word representations from the raw character stream. It relies on two mod- elling stages that are jointly learnt: convolutional network that infers a worda representation directly from the character stream, followed by a prediction stage. Models are evaluated on a POS and mor- phological tagging task for German. Ex- perimental results show that the convolu- tional network can infer meaningful word representations, while for the prediction stage, a well designed and structured strat- egy allows the model to outperform state- of-the-art results, without any feature en- gineering. 1 Introduction Most modern statistical models for natural lan- guage processing (NLP) applications are strongly or fully lexicalized, for instance part-of-speech (POS) and named entity taggers, as well as lan- guage models, and parsers. In these models, the observed word form is considered as the elemen- tary unit, while its morphological properties re- main neglected. As a result, the vocabulary ob- served on training data heavily restricts the gener- alization power of lexicalized models. Designing subword-level systems is appealing for several reasons. First, words sharing morpho- logical properties often share grammatical func- tion and meaning, and leveraging that information can yield improved word representations. Sec- ond, a subword-level analysis can address the out- of-vocabulary issue i.e the fact that word-level models fail to meaningfully process unseen word forms. This allows a better processing of morpho- logically rich languages in which there is a com- binatorial explosion of word forms, most of which are not observed during training. Finally, using subword units could allow processing of noisy text such as user-generated content on the Web, where abbreviations, slang usage and spelling mistakes cause the number of word types to explode. This work investigates models that do not rely on a fixed vocabulary to make a linguistic predic- tion. Our main focus in this paper is POS tag- ging, yet the proposed approach could be applied to a wide variety of language processing tasks. Our main contribution is to show that neural net- works can successfully learn unlexicalized mod- els that infer a useful word representation from the character stream. This approach achieves state of-the-art performance on a German POS tagging task. This task is difficult because German is a morphologically rich language1, as reflected by the large number of morphological tags (255) in our study, yielding a grand total of more than 600 POS+MORPH tags. An aggravating factor is that these morphological categories are overtly marked by a handful of highly ambiguous inflec- tion marks (suffixes). We therefore believe that this case study is well suited to assess both the rep- resentation and prediction power of our …

Content truncated to 500 words