Unsupervised Neural Text Simplification

Unsupervised Neural Text Simplification Sai Surya† Abhijit Mishra‡ Anirban Laha‡ Parag Jain‡ Karthik Sankaranarayanan‡ †IIT Kharagpur, India ‡IBM Research subramanyamdvss@gmail.com {abhijimi,anirlaha,pajain34,kartsank}@in.ibm.com Abstract The paper presents a first attempt towards un- supervised neural text simplification that re- lies only on unlabeled text corpora. The core framework is composed of a shared encoder and a pair of attentional-decoders, crucially as- sisted by discrimination-based losses and de- noising. The framework is trained using unla- beled text collected from en-Wikipedia dump. Our analysis (both quantitative and qualita- tive involving human evaluators) on public test data shows that the proposed model can perform text-simplification at both lexical and syntactic levels, competitive to existing super- vised methods. It also outperforms viable un- supervised baselines. Adding a few labeled pairs helps improve the performance further. 1 Introduction Text Simplification (TS) deals with transforming the original text into simplified variants to increase its readability and understandability. TS is an im- portant task in computational linguistics, and has numerous use-cases in fields of education technol- ogy, targeted content creation, language learning, where producing variants of the text with vary- ing degree of simplicity is desired. TS systems are typically designed to simplify from two differ- ent linguistic aspects: (a) Lexical aspect, by re- placing complex words in the input with simpler synonyms (Devlin, 1998; Candido Jr et al., 2009; Yatskar et al., 2010; Biran et al., 2011; Glavaˇs and ˇStajner, 2015), and (b) Syntactic aspect, by altering the inherent hierarchical structure of the sentences (Chandrasekar and Srinivas, 1997; Can- ning and Tait, 1999; Siddharthan, 2006; Filippova and Strube, 2008; Brouwers et al., 2014). From the perspective of sentence construction, sentence simplification can be thought to be a form of text-transformation that involves three major types of operations such as (a) splitting (Siddharthan, 2006; Petersen and Ostendorf, 2007; Narayan and Gardent, 2014) (b) deletion/compression (Knight and Marcu, 2002; Clarke and Lapata, 2006; Fil- ippova and Strube, 2008; Rush et al., 2015; Filip- pova et al., 2015), and (c) paraphrasing (Specia, 2010; Coster and Kauchak, 2011; Wubben et al., 2012; Wang et al., 2016; Nisioi et al., 2017). Most of the current TS systems require large- scale parallel corpora for training (except for sys- tems like Glavaˇs and ˇStajner (2015) that performs only lexical-simplification), which is a major im- pediment in scaling to newer languages, use-cases, domains and output styles for which such large- scale parallel data do not exist. In fact, one of the popular corpus for TS in English language, i.e., the Wikipedia-SimpleWikipedia aligned dataset has been prone to noise (mis-aligned instances) and inadequacy (i.e., instances having non-simplified targets) (Xu et al., 2015; ˇStajner et al., 2015), lead- ing to noisy supervised models (Wubben et al., 2012). While creation of better datasets (such as, Newsela by Xu et al. (2015)) can always help, we explore the unsupervised learning paradigm which can potentially work with unlabeled datasets that are cheaper and easier to obtain. At the heart of the TS problem is the need for preservation of language semantics with …

Content truncated to 500 words