A Domain-Specific Dataset of Difficulty Ratings for German Noun Compounds in the Domains DIY, Cooking and Automotive

Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pages 4359–4367 Marseille, 11–16 May 2020 c⃝ European Language Resources Association (ELRA), licensed under CC-BY-NC 4359 A Domain-Specific Dataset of Difficulty Ratings for German Noun Compounds in the Domains DIY, Cooking and Automotive Julia Bettinger1,2, Anna H¨atty1,2, Michael Dorna1, Sabine Schulte im Walde2 1Robert Bosch GmbH, Corporate Research, Renningen, Germany 2Institute for Natural Language Processing, University of Stuttgart, Germany {julia.bettinger, schulte}@ims.uni-stuttgart.de, {anna.haetty, michael.dorna}@de.bosch.com Abstract We present a dataset with difficulty ratings for 1,030 German closed noun compounds extracted from domain-specific texts for do-it-ourself (DIY), cooking and automotive. The dataset includes two-part compounds for cooking and DIY, and two- to four-part compounds for automotive. The compounds were identified in text using the Simple Compound Splitter (Weller-Di Marco, 2017); a subset was filtered and balanced for frequency and productivity criteria as basis for manual annotation and fine-grained interpretation. This study presents the creation, the final dataset with ratings from 20 annotators and statistics over the dataset, to provide insight into the perception of domain-specific term difficulty. It is particularly striking that annotators agree on a coarse, binary distinction between easy vs. difficult domain-specific compounds but that a more fine-grained distinction of difficulty is not meaningful. We finally discuss the challenges of an annotation for difficulty, which includes both the task description as well as the selection of the data basis. Keywords: Domain-Specific Terminology, Term Difficulty, Closed Compounds 1. Introduction Domain-specific texts contain domain-relevant vocabulary, which is potentially difficult to understand for people with- out specialized knowledge about the domain. The ad- dressed audiences of domain-specific texts might also vary. Some texts are clearly written for experts while others are written especially for laypersons. It is thus important to identify the difficulty of words contained in domain- specific texts, as one of the building blocks to draw con- clusions about domain-specific text difficulty. For these reasons, we create a German closed noun com- pound dataset that was extracted from three domains: do-it- yourself (DIY), cooking and automotive. We chose closed compounds because they represent a kind of multiword ex- pression that is easy to identify in text, and we therefore do not need elaborate preprocessing for finding valid phrases. At the same time, it is highly likely for multiword ex- pressions in domain-specific texts that they carry domain- relevant meanings. For example, Justeson and Katz (1995) find that the most common form of a domain-specific term for English is a two-word noun compound. In the following, we describe the development of the domain-specific compound dataset and its annotation for difficulty. This includes the construction of domain-specific corpora to extract the compounds (section 4.), and the ac- tual creation of the compound dataset (section 5.). After evaluating the interannotator agreement (section 6.), the final gold standard is described (section 7.). We present statistics and insights about the dataset (section 8.) and dis- cuss the challenges of annotating difficulty (section 9.). 2. Related Work Detecting a lay reader’s familiarity or difficulty with domain-specific …

Content truncated to 500 words