Text Readability Classification of Textbooks of a Low-Resource Language

Copyright 2012 by Zahurul Islam, Alexander Mehler, and Rashedur Rahman 26th Pacific Asia Conference on Language,Information and Computation pages 545–553 Text Readability Classification of Textbooks of a Low-Resource Language Zahurul Islam, Alexander Mehler and Rashedur Rahman AG Texttechnology Institut f¨ur Informatik Goethe-Universit¨at Frankfurt zahurul,mehler@em.uni-frankfurt.de, kamol.sustcse@gmail.com Abstract There are many languages considered to be low-density languages, either because the population speaking the language is not very large, or because insufficient digitized text material is available in the language even though millions of people speak the language. Bangla is one of the latter ones. Readabil- ity classification is an important Natural Lan- guage Processing (NLP) application that can be used to judge the quality of documents and assist writers to locate possible prob- lems. This paper presents a readability clas- sifier of Bangla textbook documents based on information-theoretic and lexical features. The features proposed in this paper result in an F-score that is 50% higher than that for tra- ditional readability formulas. 1 Introduction The readability of a text relates to how easily hu- man readers can process and understand a text as the writer of the text intended. There are many text re- lated factors that influence the readability of a text. These factors include very simple features such as type face, font size, text vocabulary as well as com- plex features like grammatical conciseness, clarity, underlying semantics and lack of ambiguity. Nowadays, teachers, journalists, editors and other professionals who create text for a specific audience routinely check the readability of their text. Read- ability classification, then, is the task of mapping text onto a scale of readability levels. We explore the task of automatically classifying documents based on their different readability levels. As input, this function operates on various statistics relating to lex- ical and other text features. Automatic readability classification can be useful for many Natural Language Processing (NLP) appli- cations. Automatic essay grading can benefit from readability classification as a guide to how good an essay actually is. Similarly, search engines can use a readability classifier to rank its generated search results. Automatically generated documents, for ex- ample documents generated by text summarization systems or machine translation systems, tend to be error-prone and less readable. In this case, a read- ability classification system can be used to filter out documents that are less readable. The system can also be used to evaluate machine translation output. A document of higher readability tends to be better than a document that belongs to a lower readability class. Research in the field of readability classification started in 1920. English is the dominating language in this field although much research has been done for other languages like German, French, Chinese and so on. These languages are considered as high- density languages as many language resources and tools are available for them. However, many lan- guages are considered to be low-density languages, either because the population speaking the language is not very large or because insufficient digitized text material is available for the language …

Content truncated to 500 words