On- and Off-Topic Classification and Semantic Annotation of User-Generated Software Requirements

Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1807–1816, Austin, Texas, November 1-5, 2016. c⃝2016 Association for Computational Linguistics On- and Off-Topic Classification and Semantic Annotation of User-Generated Software Requirements Markus Dollmann and Michaela Geierhos Heinz Nixdorf Institute University of Paderborn F¨urstenallee 11, 33102 Paderborn, Germany {dollmann|geierhos}@hni.upb.de Abstract Users prefer natural language software re- quirements because of their usability and ac- cessibility. When they describe their wishes for software development, they often provide off-topic information. We therefore present REaCT1, an automated approach for identify- ing and semantically annotating the on-topic parts of requirement descriptions. It is de- signed to support requirement engineers in the elicitation process on detecting and analyzing requirements in user-generated content. Since no lexical resources with domain-specific in- formation about requirements are available, we created a corpus of requirements writ- ten in controlled language by instructed users and uncontrolled language by uninstructed users. We annotated these requirements re- garding predicate-argument structures, con- ditions, priorities, motivations and semantic roles and used this information to train clas- sifiers for information extraction purposes. REaCT achieves an accuracy of 92% for the on- and off-topic classification task and an F1- measure of 72% for the semantic annotation. 1 Introduction “Requirements are what the software product, or hardware product, or service, or whatever you in- tend to build, is meant to do and to be” (Robert- son and Robertson, 2012). This intuitive descrip- tion of requirements has one disadvantage. It is as vague as a requirement that is written by an un- trained user. More generally, functional require- ments define what a product, system or process, or 1Requirements Extraction and Classification Tool a part of it is meant to do (Robertson and Robert- son, 2012; Vlas and Robinson, 2011). Due to its ex- pressiveness, natural language (NL) became a pop- ular medium of communication between users and developers during the requirement elicitation pro- cess (de Almeida Ferreira and da Silva, 2012; Mich et al., 2004). Especially in large ICT projects, re- quirements, wishes, and ideas of up to thousands of different users have to be grasped (Castro-Herrera et al., 2009). For this purpose, requirement en- gineers collect their data, look for project-relevant concepts and summarize the identified technical fea- tures. However, this hand-crafted aggregation and translation process from NL to formal specifica- tions is error-prone (Goldin and Berry, 1994). Since people are getting tired and unfocused during this monotonous work, the risk of information loss in- creases. Hence, this process should be automated as far as possible to support requirement engineers. In this paper, we introduce our approach to iden- tify and annotate requirements in user-generated content. We acquired feature requests for open source software from SourceForge2, specified by (potential) users of the software. We divided these requests into off-topic information and (on-topic) requirements to train a binary text classifier. This allows an automated identification of new require- ments in user-generated content. In addition, we col- lected requirements in controlled language from the NFR corpus3 and from …

Content truncated to 500 words