Automated Error Correction and Validation for POS Tagging of Hindi

Automated Error Correction and Validation for POS Tagging of Hindi Sachi Angle MIT, Manipal sachiangle@gmail.com Pruthwik Mishra IIIT, Hyderabad pruthwikmishra@gmail.com Dipti Misra Sharma IIIT, Hyderabad dipti@iiit.ac.in Abstract The Part-Of-Speech tag of a word can pro- vide crucial information for a large num- ber of tasks, and so, it is of utmost im- portance that the POS tagged data is ac- curate. However, manually checking the data is a tedious and time consuming task. Thus, there is a need for an Automatic Er- ror Correction and Validation model for any POS Tagged Data. In this paper, we work towards achieving the aforemen- tioned goal for Hindi POS Tagging. This is achieved by using an ensemble model consisting of three POS Tagging Models. Based on the predictions made by the three models, and the POS tag present in the dataset, the ensemble model predicts the presence of an error. The POS tagging models explored were the Hidden Markov Model, Support Vector Machine, Condi- tional Random Fields, Long Short Term Memory (LSTM) Networks, Bidirectional LSTM Networks, and Logistic Regression. A Fully Connected Neural Network was used to build the ensemble model, and it achieved an accuracy of 94.02%. 1 Introduction Part-Of-Speech (POS) tagging refers to catego- rizing words into the POS categories (e.g. noun, verb, or conjunction) they belong to. It forms an essential component of a large number of Nat- ural Language Processing tasks, ranging from speech modeling, where they affect the pronun- ciation, to Information Retrieval, where they are helpful in stemming. Although extensive work has been done on POS tagging of English, there is still a lot of scope for advancement for the same with respect to Indian Languages. Common models used for POS tagging in- clude the Hidden Markov Model (HMM), Long- Short Term Memory (LSTM) Networks, and Conditional Random Fields(CRF). These mod- els can efficiently tag the data as they take into account the previous states of data real- ized through words and tags. This greatly in- creases the accuracy with which the data is tagged. Significant work has been undertaken for Hindi POS Tagging, however not enough has been done yet to build an Automatic Er- ror Correction and Validation model. Models explored to achieve this involve a lot of manual tagging and checking of data. Therefore, there is a need to build a model that reduces human effort. The method to be followed includes the predictions of three models for a particular con- text, and based on these predictions, check for the presence of an error. This model will au- tomate error checking, thus ending the need for manual checking. 2 Literature Survey Traditionally, rule-based tagging, stochastic tag- ging, and transformation-based tagging have been used for POS Tagging. Rule-based tag- gers generally involve a large database of hand- written disambiguation rules. On the other hand, a Stochastic Tagger relies on available data and on probability to resolve ambiguities. PACLIC 32 11 32nd Pacific Asia Conference on Language, Information and Computation Hong Kong, 1-3 December 2018 Copyright …

Content truncated to 500 words