Unsupervised and Lightly Supervised Part-of-Speech Tagging Using Recurrent Neural Networks

Unsupervised and Lightly Supervised Part-of-Speech Tagging Using Recurrent Neural Networks Othman Zennaki 1,2 Nasredine Semmar 1 1CEA, LIST, Vision and Content Engineering Laboratory, Gif-sur-Yvette, France {othman.zennaki, nasredine.semmar}@cea.fr 2Laboratory of Informatics of Grenoble, Univ. Grenoble-Alpes, Grenoble, France laurent.besacier@imag.fr Laurent Besacier 2 Abstract In this paper, we propose a novel approach to induce automatically a Part-Of-Speech (POS) tagger for resource-poor languages (languages that have no labeled training data). This ap- proach is based on cross-language projection of linguistic annotations from parallel cor- pora without the use of word alignment infor- mation. Our approach does not assume any knowledge about foreign languages, making it applicable to a wide range of resource-poor languages. We use Recurrent Neural Net- works (RNNs) as multilingual analysis tool. Our approach combined with a basic cross- lingual projection method (using word align- ment information) achieves comparable re- sults to the state-of-the-art. We also use our approach in a weakly supervised context, and it shows an excellent potential for very low- resource settings (less than 1k training utter- ances). 1 Introduction Nowadays, Natural Language Processing (NLP) tools (part-of-speech tagger, sense tagger, syntactic parser, named entity recognizer, semantic role la- beler, etc.) with the best performance are those built using supervised learning approaches for resource- rich languages (where manually annotated corpora are available) such as English, French, German, Chi- nese and Arabic. However, for a large number of resource-poor languages, annotated corpora do not exist. Their manual construction is labor intensive and very expensive, making supervised approaches not feasible. The availability of parallel corpora has recently led to several strands of research work exploring the use of unsupervised approaches based on lin- guistic annotations projection from the (resource- rich) source language to the (under-resourced) tar- get language. The goal of cross-language projec- tion is, on the one hand, to provide all languages with linguistic annotations, and on the other hand, to automatically induce NLP tools for these lan- guages. Unfortunately, the state-of-the-art in un- supervised methods, is still quite far from super- vised learning approaches. For example, Petrov et al. (2012) obtained an average accuracy of 95.2% for 22 resource-rich languages supervised POS tag- gers, while the state-of-the-art in the unsupervised POS taggers achieved by Das and Petrov (2011) and Duong et al. (2013) with an average accuracy reaches only 83.4% on 8 European languages. Sec- tion 2 presents a brief overview of related work. In this paper, we first adapt a similar method than the one of Duong et al. (2013) 1, to build an unsu- pervised POS tagger based on a simple cross-lingual projection (Section 3.1). Next, we explore the possi- bility of using a recurrent neural network (RNN) to induce multilingual NLP tools, without using word alignment information. To show the potential of our approach, we firstly investigate POS tagging. In our approach, a parallel corpus between a resource-rich language (having a POS tagger) and a lower-resourced language is used to extract a com- mon words representation (cross-lingual words rep- resentation) based only on sentence level alignment. This …

Content truncated to 500 words