A Detailed Evaluation of Neural Sequence-to-Sequence Models for In-domain and Cross-domain Text Simplification

A Detailed Evaluation of Neural Sequence-to-Sequence Models for In-domain and Cross-domain Text Simplification Sanja ˇStajner1,∗, Sergiu Nisioi2,∗ 1 DWS Research Group, University of Mannheim, Germany 2 Human Language Technologies Research Center, University of Bucharest, Romania sanja@informatik.uni-mannheim.de, sergiu.nisioi@fmi.unibuc.ro Abstract We present a detailed evaluation and analysis of neural sequence-to-sequence models for text simplification on two distinct datasets: Wikipedia and Newsela. We employ both human and automatic evaluation to investigate the capacity of neural models to generalize across corpora, and we highlight challenges that these models face when tested on a different genre. Furthermore, we establish a strong baseline on the Newsela dataset and show that a simple neural architecture can be efficiently used for in-domain and cross-domain text simplification. Keywords: neural text simplification, sequence-to-sequence models, evaluation 1. Introduction The aim of text simplification (TS) is to transform given texts into their syntactically and/or lexically simpler vari- ants which are more understandable for the target popula- tion (e.g. children, non-native speakers, people with low literacy levels, or people with various kinds of cognitive or reading impairments). It is usually applied on the sentence level and encompasses three major simplification opera- tions: sentence splitting, deletions, and lexical paraphrases (Xu et al., 2016). In recent years, the problem of automated sentence simpli- fication has often been addressed as the monolingual ma- chine translation (MT) task of translating from original to simple sentences. The MT models used were, however, adapted to the specificities of the TS task, e.g. by adding phrasal deletions to the standard phrase-based MT model (Coster and Kauchak, 2011) to account for a common sen- tence shortening and phrasal deletions in TS, or by rerank- ing the output of the phrase-based MT model (Wubben et al., 2012), since in the standard phrase-based MT model applied on TS, the first hypothesis tends to leave the in- put unchanged (Specia, 2010; Coster and Kauchak, 2011). Until recently, the state of the art for automated text sim- plification was the syntax-based machine translation sys- tem (SBMT) with specific optimizations for TS (Xu et al., 2016), such as the use of a large paraphrase database (PPDB) to boost the coverage of the phrasal simplifications, and the use of SARI, a tuning metric that particularly re- wards simplicity (Xu et al., 2016). Following more recent advancements in machine trans- lation using neural networks, we proposed a neural text simplification system, which significantly outperformed the state of the art on various evaluation metrics for the Wikipedia dataset (Nisioi et al., 2017). Our model was constructed as a vanilla encoder-decoder architecture with global attention and input feeding. More recently, a neu- ral network model fine-tuned using reinforcement learn- ing (Zhang and Lapata, 2017) has been proposed for text simplification, the authors reporting several improvements over the previous systems. Given that they were proposed around the same time, the two neural TS models have not yet been directly compared. One commonly raised issue with most supervised TS sys- tems is the usage of English Wikipedia – Simple English Wikipedia (EW–SEW) …

Content truncated to 500 words