Attention-based Domain Adaption Using Transfer Learning for Part-of-Speech Tagging: An Experiment on the Hindi language

Attention-based Domain Adaption Using Transfer Learning for Part-of-Speech Tagging: An Experiment on the Hindi Language Rajesh Kumar Mundotiya, Vikrant Kumar, Arpit Mehta and Anil Kumar Singh Department of Computer Science and Engineering, IIT(BHU), Varanasi, India {rajeshkm.rs.cse16, vikrantkumar.cse18}@iitbhu.ac.in, {arpitmehta.cse18, aksingh.cse}@iitbhu.ac.in Abstract Part-of-Speech (POS) tagging is considered a preliminary task for parsing any language, which in turn is required for many Natural Language Processing (NLP) applications. Ex- isting work on the Hindi language for this task reported results on either the General or the News domain from the Hindi-Urdu Treebank that relied on a reasonably large annotated corpus. Since the Hindi datasets of the Dis- ease and the Tourism domain have less anno- tated corpus, using domain adaptation seems to be a promising approach. In this paper, we describe an attention-based model with self- attention as well as monotonic chunk-wise attention, which successfully leverage syn- tactic relations through training on a small dataset. The accuracy of the Hindi Dis- ease dataset performed by the attention-based model using transfer learning is 93.86%, an improvement on the baseline model (93.64%). In terms of F1-score, however, the base- line model (93.65%) seems to do better than the monotonic-chunk-wise attention model (94.05%). 1 Introduction Deep learning has been consistently providing promising results on a large variety of language pro- cessing problems. Textual processing includes di- verse applications of NLP such as text classification, dialect identification and classification, sequence la- belling problems (such as Named Entity Recogni- tion and Extraction, Chunking and POS tagging) and machine translation. However, for performance improvement obtained on the preliminary NLP tasks – POS tagging and Chunking – especially under a low resource sce- nario, Recurrent Neural Network (RNN) and Con- volutional Neural Network (CNN) have been used more. An efficient way of information modeling by Gated Recurrent (GRU) and Long Short Term Mem- ory (LSTM), a variant of RNN, has also been tried. Earlier work on POS tagger for canonical Hindi text achieved considerable results of about 97.10% on Universal Dependency dataset (Plank et al., 2016), which belongs to a single domain. The per- formance reduces radically after deploying this ex- isting trained model to a different domain-specific data or out-of-domain data. Domain-specific data such as Tourism and Disease has its own distribu- tions and having a minimal amount of annotated dataset, considered as low resources, which also causes an Out-of-vocabulary (OOV) words issue. OOV is a major problem in low resources text pro- cessing, faced while training a model on one domain of a language and trying it to another domain of the same language. This problem is partly countered by incorporating character level information into the model. Lately, Transfer Learning has been shown to en- hance the performance of the model by transferring learned features (general features as well as domain- specific features) which were obtained during train- ing the model. The general features are transferred to the target domain through an initializer or feature extractor. These methods are beneficial as they ben- efit from the pre-trained model via …

Content truncated to 500 words