Review improvement by requirements classification at Mercedes-Benz: Limits of empirical studies in educational environments

Reviews are the most common way to ensure quality in natural language (NL) requirements specifications. But with increasing size (up to 3,000 pages in the automotive domain) and complexity of the specification documents, the review task tends to be less effective. To improve the review task for large documents, one possible solution is the `topic landscape´. The idea of this approach is to introduce a pre-classification and clustering of requirements according to topics. In a first empirical study with eight students, we analyzed the effectiveness of the topic landscape approach. During this study, we encountered a general limitation of experiments with large requirements specifications, especially in external environments like universities. Industries have a strong demand on new approaches and methods to deal with large specifications. However, there is a growing gap between the number of requirements that can be examined during an empirical study and the number of requirements required to ensure results that are valid for real requirements specifications. This paper describes the conducted empirical study in detail and shows recognized problems concerning the limits of educational environments.