Some validity concerns in nonexperimental program evaluation: a case study

It is acknowledged that educational research and evaluation is fraught with credibility and validity concerns, yet available research methodology literature has focused on "experimentally" designed programs although the greatest evaluation challenges are presented by non- "experimentally" designed programs that are already in place and which have been institutionalized. Acknowledging the fact that the nature of educational research is applied, heterogeneous, and multidimensional, the primary purpose of this paper is to provide a case-study insight into evaluating the longitudinal impact of a non-"experimentally" designed student cohorting strategy, which has been institutionalized in the Texas A&M University freshman-engineering program. Originally introduced as a pilot project by the Foundation Coalition Program, this strategy has grouped freshman-engineering students by enrolling them together in clusters of common, required, classes. This paper analyzes and discusses the challenges of this evaluation task: i.e., applying appropriate ex post facto strategies; controlling for non-"experimental" treatment of groups; and identifying valid success indicators. Its objective is to discuss the validity concerns encountered from an applied methodological perspective. A full presentation of the evaluated program\´s outcomes is not within the scope of this paper; however the evaluation study is ongoing and a number of papers addressing program outcomes are in preparation.