Given these difficult problems with a classic experimental design, many program leaders and program evaluators may have to adopt alternative designs that are less elegant but more practical. In some cases, leaders and evaluators have restricted their assessment to outcome measures. They determine the level of performance achieved by a group of coaching clients and use this information to determine the relative success of the program being evaluated. This type of information is subject to many misinterpretations and abuses, though it is the most common evaluation design being used in contemporary organizations.
The information is flawed even when a comparison is drawn with coaching programs in other divisions of the organization or in other organizations. One doesn’t know if differences in performance of the coaching clients can be attributed to the coaching program being reviewed or to the entering characteristics of the clients. Did clients in the alpha division or at the alpha organization do better than clients in the beta division or at the beta organization because alpha clients were already better trained or working at a higher level than beta clients before they even entered the coaching program?
This confounding effect is prevalent in many of the current evidence-based initiatives and even the ROI investigations that call for clients to perform at a certain level without any consideration being given to their level of performance upon entering the coaching program. In order to be fair in the assessment of a coaching program’s effectiveness, one must at the very least perform a “value-added” assessment. This type of assessment requires that a coaching client’s performance be measured when they first enter the coaching program and again when they “graduate” from the program to determine the “value” that has been added, or more specifically the improvement in performance that has been observed and recorded.
Fortunately, there are ways in which to assess program outcomes accurately and fairly, without having to engage a pure experimental design that may be neither feasible nor ethical. Two of the most widely respected authorities in the field of program evaluation, Donald Campbell and Julian Stanley, described a set of “quasi-experimental” designs that allow one to modify some of the conditions of a traditional experimental design without sacrificing the clarity of results obtained. Campbell and Stanley’s brief monograph on experimental and quasi-experimental designs is a classic in the field. Any program evaluator who wishes to design an outcome determination evaluation should consult this monograph. Three of the most widely used of these quasi-experimental designs are “time series,” “nonequivalent control group design” and “rotational/counterbalanced design.”
Download Article 1K Club
