Comparing and Developing Tools to Measure the Readability of Domain-Specific Texts
9026Authors
nan
Abstract
AbstractThe readability of a digital text can influence people’s ability to learn new things about a range topics from digital resources (e.g., Wikipedia, WebMD). Readability also impacts search rankings, and is used to evaluate the performance of NLP systems. Despite this, we lack a thorough understanding of how to validly measure readability at scale, especially for domain-specific texts. In this work, we present a comparison of the validity of well-known readability measures and introduce a novel approach, Smart Cloze, which is designed to address shortcomings of existing measures. We compare these approaches across four different corpora: crowdworker-generated stories, Wikipedia articles, security and privacy advice, and health information. On these corpora, we evaluate the convergent and content validity of each measure, and detail tradeoffs in score precision, domain-specificity, and participant burden. These results provide a foundation for more accurate readability measurements and better evaluation of new natural-language-processing systems and tools.
Keywords
Full Content
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4831–4842, Hong Kong, China, November 3–7, 2019. c⃝2019 Association for Computational Linguistics 4831 C o m p ari n g a n d De vel o pi n g To ols t o Me as ure t he Re a d a bilit y of D o m ai n- S peci fic Te xts Eliss a M. Re d miles 1 , Lis a M asz kie wicz1 , E mil y H w a n g1 , D hr u v K uc h h al2 , E verest Li u 1 , Mir ai d a M or ales3 , De nis Pes k o v1 , S u d h a R a o1 , R oc k Ste ve ns 1 , Kristi n a Gli g ori´c 4 , Se a n Kr oss5 , Mic helle L. M az ure k1 , a n d H al D a u m ´e III 1, 6 1 U ni versit y of 2 Ge or gia Tec hMar yla n d { e r e d m i l e s , m m a z u r e k , h a l } @ c s . u m d . e d u 3 R ut gers U ni versit y 4 E P F L 5 U C S D 6 Micr os oft Researc h A bstr act T he rea da bilit y of a di gital te xt ca n i n fl u- e nce pe o ple’s a bilit y t o lear n ne w t hi n gs a b o ut a ra n ge t o pics fr o m di gital res o urces (e. g., Wi ki pe dia, We b M D). Rea da bilit y als o i m pacts searc h ra n ki n gs, a n d is use d t o e val uate t he perf or ma nce of N L P s yste ms. Des pite t his, we lac k a t h or o u g h u n dersta n di n g of h o w t o vali dl y meas ure rea da bilit y at scale, es peciall y f or d o mai n-s peci fic te xts. I n t his w or k, t he vali dit y ofwe prese nt a c o m paris o n of well- k n o w n rea da bilit y mea- s ures a n d i ntr o d uce a n o vel a p pr oac h, S m art Cl oze , w hic h is desi g ne d t o a d dress s h ort- c o mi n gs of …
Content truncated to 500 words
Labeled Text
AbstractThe readability of a digital text can influence people’s ability to learn new things about a range topics from digital resources (e.g., Wikipedia, WebMD). Readability also impacts search rankings, and is used to evaluate the performance of NLP systems. Despite this, we lack a thorough understanding of how to validly measure readability at scale, especially for domain-specific texts. In this work, we present a comparison of the validity of well-known readability measures and introduce a novel approach, Smart Cloze, which is designed to address shortcomings of existing measures. We compare these approaches across four different corpora: crowdworker-generated stories, Wikipedia articles, security and privacy advice, and health information. On these corpora, we evaluate the convergent and content validity of each measure, and detail tradeoffs in score precision, domain-specificity, and participant burden. These results provide a foundation for more accurate readability measurements and better evaluation of new natural-language-processing systems and tools.
Publication Details
- Paper ID:
- 9026
- Year:
- 2019
- Publication:
- TIPSTER
- Publication Type:
- conf
- DOI:
- nan