Improving Text Simplification Language Modeling Using Unsimplified Text Data

Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages 1537–1546, Sofia, Bulgaria, August 4-9 2013. c⃝2013 Association for Computational Linguistics Improving Text Simplification Language Modeling Using Unsimplified Text Data David Kauchak Middlebury College Middlebury, VT 05753 dkauchak@middlebury.edu Abstract In this paper we examine language mod- eling for text simplification. Unlike some text-to-text translation tasks, text simplifi- cation is a monolingual translation task al- lowing for text in both the input and out- put domain to be used for training the lan- guage model. We explore the relation- ship between normal English and simpli- fied English and compare language mod- els trained on varying amounts of text from each. We evaluate the models intrin- sically with perplexity and extrinsically on the lexical simplification task from Se- mEval 2012. We find that a combined model using both simplified and normal English data achieves a 23% improvement in perplexity and a 24% improvement on the lexical simplification task over a model trained only on simple data. Post-hoc anal- ysis shows that the additional unsimplified data provides better coverage for unseen and rare n-grams. 1 Introduction An important component of many text-to-text translation systems is the language model which predicts the likelihood of a text sequence being produced in the output language. In some problem domains, such as machine translation, the trans- lation is between two distinct languages and the language model can only be trained on data in the output language. However, some problem do- mains (e.g. text compression, text simplification and summarization) can be viewed as monolingual translation tasks, translating between text varia- tions within a single language. In these monolin- gual problems, text could be used from both the input and output domain to train a language model. In this paper, we investigate this possibility for text simplification where both simplified English text and normal English text are available for training a simple English language model. Table 1 shows the n-gram overlap proportions in a sentence aligned data set of 137K sentence pairs from aligning Simple English Wikipedia and English Wikipedia articles (Coster and Kauchak, 2011a).1 The data highlights two conflicting views: does the benefit of additional data out- weigh the problem of the source of the data? Throughout the rest of this paper we refer to sentences/articles/text from English Wikipedia as normal and sentences/articles/text from Simple English Wikipedia as simple. On the one hand, there is a strong correspon- dence between the simple and normal data. At the word level 96% of the simple words are found in the normal corpus and even for n-grams as large as 5, more than half of the n-grams can be found in the normal text. In addition, the normal text does represent English text and contains many n-grams not seen in the simple corpus. This extra informa- tion may help with data sparsity, providing better estimates for rare and unseen n-grams. On the other hand, there is still only modest overlap between the sentences for longer n-grams, particularly given …

Content truncated to 500 words