Attention-based BLSTM-CRF Architecture for Mongolian Named Entity Recognition

Attention-based BLSTM-CRF Architecture for Mongolian Named Entity Recognition Yuzhu Xiong Minghua Nuo* College of Computer Science College of Computer Science Inner Mongolia University Inner Mongolia University Huhhot 010021, China Huhhot 010021, China evilbear@live.cn nuominghua@163.com Abstract Adding external information from unlabeled data to Named Entity Recognition task is the direction of research for solving the scarcity of labeled data. In this issue, adding pre-trained context embeddings as external information from language model is the state-of-the-art technology. However, it has combined external information to the model with simple concatenation, which assumed the same contribution of both internal and external information. While with the use of pre-trained language models, model parameters were not be updated during the iterative process of named entity recognition model. The language model cannot fully capture the semantic and syntactic roles of the contextual words in the labeled data. In view of that case, this paper proposed an improved approach by using pre-trained context embeddings based on attention mechanism. The attention mechanism layer can dynamically balance the difference between the internal information learned from bidirectional Long Short-term Memory model and external information from neural language model. Experimental results have shown that our model has achieved substantial improvement over previous one in Mongolian Named Entity Recognition task. 1 Introduction Named entity recognition (NER) is the basic step of many natural language processing (NLP) tasks, such as Information Extraction and Machine Translation. NER's recognition precision will directly affect subsequent NLP tasks. Mongolian has a wide range of users in China, Mongolia and Russia. However, it is a low-resource language, Mongolian NLP resources are very rare. The research on Mongolian NLP is at its initial stage. Researchers used Conditional Random Field (CRF) to predict Mongolian NER label and only did initial work on it. In terms of the NER task, we need to label the sentences with named entity tags. For example, Figure 1 gives a sentence with organization name (ORG) and person name (PER). Figure 1: Example of traditional Mongolian script and named entity tags In Mongolian corpus, data sparseness problem is more serious, due to the lexical features of Mongolian and homomorphic characters with different pronunciation, words look same but pronu- nciation is different in the corpus. Sometimes, it occurs to spell mistake in Mongolian corpus which is not consistent with the coding rule due to the PACLIC 32 761 32nd Pacific Asia Conference on Language, Information and Computation Hong Kong, 1-3 December 2018 Copyright 2018 by the authors reason of keyboard operators' dialect. For example, the words "oyun" and "uyun" are expressed in Latin Mongolian, and they are both the word "wisdom" in English. They are the same word rather than a traditional synonym. To some extent, this phenomenon can lead to data sparsity. For further details, we have made word frequency statistics on web corpus, and there are a large number of low frequency words, some of which are misspelling of high frequency words. They have the same context, and the neural LM can learn this representation of …

Content truncated to 500 words