uniblock: Scoring and Filtering Corpus with Unicode Block Information

Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 1324–1329, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 1324 uniblock: Scoring and Filtering Corpus with Unicode Block Information Yingbo Gao Weiyue Wang Hermann Ney Human Language Technology and Pattern Recognition Group Computer Science Department RWTH Aachen University D-52056 Aachen, Germany <surname>@i6.informatik.rwth-aachen.de Abstract The preprocessing pipelines in Natural Lan- guage Processing usually involve a step of re- moving sentences consisted of illegal charac- ters. The definition of illegal characters and the specific removal strategy depend on the task, language, domain, etc, which often lead to tiresome and repetitive scripting of rules. In this paper, we introduce a simple statistical method, uniblock1, to overcome this prob- lem. For each sentence, uniblock gener- ates a fixed-size feature vector using Unicode block information of the characters. A Gaus- sian mixture model is then estimated on some clean corpus using variational inference. The learned model can then be used to score sen- tences and filter corpus. We present exper- imental results on Sentiment Analysis, Lan- guage Modeling and Machine Translation, and show the simplicity and effectiveness of our method. 1 Introduction Identification and removal of sentences with ille- gal characters is a common heuristic in the prepro- cessing pipelines in Natural Language Processing (NLP). While it has benefits of controlling the vo- cabulary size and dropping noisy data, it is often a tedious work to come up with appropriate rules and removal strategies when facing with different tasks, languages and domains. The lack of clear definition of what is illegal exacerbates the prob- lem. For example, modern Chinese text may allow characters such as: traditional and simplified Chi- nese characters, special punctuation marks, full- width characters, emojis, mathematical symbols, Latin characters, currency symbols, scientific no- tations, etc. As a result, scripting robust rules often requires a considerable amount of time and effort. 1The source code is available at https://github. com/ringoreality/uniblock In this paper, we introduce a simple statistical method, uniblock, to address the problem. The motivation of our approach is straightforward - good rules may vary greatly depending on the situ- ation, therefore instead of designing rules by hand, we use “rules” defined by data. We assume that some clean corpus is available, in which all the characters are deemed legal and the character dis- tributions are similar to the test cases. It is possi- ble to learn a probabilistic model, which describes the clean corpus and assigns scores to sentences in another corpus. The scores can further be used for filtering. Since the legalness of characters is in question, Unicode block information is a natu- ral choice for obtaining feature vectors. Note that, by designing alternative feature vectors, one can potentially adapt uniblock to implement other corpus filtering heuristics. We develop uniblock mainly for Machine Translation (MT). It can also be easily applied to other NLP tasks. We present experimental results on Sentiment Analysis (SA), Language Modeling …

Content truncated to 500 words