Benchmarking Lexical Simplification Systems

Benchmarking Lexical Simplification Systems Gustavo H. Paetzold, Lucia Specia University of Sheffield Western Bank, Sheffield ghpaetzold1@sheffield.ac.uk, l.specia@sheffield.ac.uk Abstract Lexical Simplification is the task of replacing complex words in a text with simpler alternatives. A variety of strategies have been devised for this challenge, yet there has been little effort in comparing their performance. In this contribution, we present a benchmarking of several Lexical Simplification systems. By combining resources created in previous work with automatic spelling and inflection correction techniques, we introduce BenchLS: a new evaluation dataset for the task. Using BenchLS, we evaluate the performance of solutions for various steps in the typical Lexical Simplification pipeline, both individually and jointly. This is the first time Lexical Simplification systems are compared in such fashion on the same data, and the findings introduce many contributions to the field, revealing several interesting properties of the systems evaluated. Keywords: Lexical Simplification, Text Simplification, Evaluation Dataset 1. Introduction The goal of a Lexical Simplification (LS) system is to re- place the complex words in a text with simpler alternatives, without compromising its meaning or grammaticality. The LS task is often addressed as the series of steps in Figure 1, as introduced by (Devlin and Tait, 1998). Their work has inspired others to conceive new LS solutions for the aphasic (Carroll et al., 1998), dyslexic (Bott et al., 2012), illiterate (Watanabe et al., 2009), non-native English speakers (Paet- zold, 2013; Paetzold and Specia, 2013), children (Kajiwara et al., 2013) and others. Figure 1: Lexical Simplification Pipeline Although various LS systems can be found in the litera- ture, very little effort has been made to compare their per- formance. Apart from the work of (Shardlow, 2013) and (Specia et al., 2012), which provide brief benchmarkings of Complex Word Identification and Substitution Ranking, respectively, no other comparisons have been reported. To address this limitation, we present a systematic bench- marking of LS systems. We innovate by comparing the per- formance of not only systems in their entirety, but also of system components individually, such as Substitution Gen- eration, Selection and Ranking approaches. In addition, we introduce BenchLS, a new dataset for the task. In the fol- lowing Sections, we describe our dataset and present our experiments. 2. BenchLS: A New Dataset To create our dataset we combined two resources: the LexMTurk (Horn et al., 2014) and LSeval (De Belder and Moens, 2012) datasets. The instances in both datasets, 929 in total, contain a sentence, a target complex word, and sev- eral candidate substitutions ranked according to their sim- plicity. The candidates in both datasets were suggested and ranked by English speakers from the U.S. To increase its reliability, we applied the following corrections over each instance of our dataset: 1. Spelling Filtering: We discard any misspelled can- didates using Norvig’s algorithm1. We trained our spelling model over the News Crawl2 corpus. 2. Inflection Correction: We inflected all candidates to the tense of the target word using the Text Adorning module of LEXenstein (Paetzold and Specia, 2015; Burns, 2013). …

Content truncated to 500 words