Large-Scale, External Evaluations of Program Impacts vs. Reliance on Program-Based Evaluation Capacity to Assess Participant Outcomes.

Or

What role can evaluation play in supporting our grantees to offer quality programs?

By David E.K. Hunter
(David Hunter served as director of Evaluation and Knowledge Development at the Edna McConnell Clark Foundation from 1998 to 2006.)

Presented at the September 24th, 2003 NYC Youth Funders Network Meeting - A Question of Quality: How Do We Know What Works in After-School?

What does it mean to claim that something “works” in after-school? Generally speaking, there are two schools of thought on this question. One holds that since kids tend to get in trouble between 3pm and 6pm, after-school programs need do nothing more than keep them safe between the end of the school day and the time their parent(s) get(s) home. The other position is that this is not enough. In addition to being kept safe, kids should derive some other, positive benefits from after-school programming. I hold with the latter view, and what I will say today builds on this premise.

Recent events have made today’s topic extraordinarily timely as well as inherently important. Earlier this year, after the release of Mathematica’s first year report of the 3-year evaluation of the 21st Century Community Learning Centers showing no evidence of improved academic outcomes for participants, the Bush administration announced plans to cut federal funding for this program by $400 million–a full 40% of the $1 billion that had been dedicated to it. In the resulting uproar several troubling things happened:

• The Mathematica study was subjected to intense criticism by leading youth development advocates.
• These advocates, in lobbying to have the funding restored, adopted a dismissive stance toward the use of evaluation data for setting funding priorities.
• Members of the evaluation profession also were drawn into the fray, with some very publicly (and in uncharacteristically dogmatic and harsh language) attacking the methods and validity of the Mathematica study, and others rising to Mathematica’s defense.
• One group, The After-School Corporation (TASC), actively worked to contrast the evaluation methods used by Policy Studies Associates (PSA) to evaluate TASC with those used by Mathematica on the 21st Century evaluation, and further asserted that the TASC evaluation in fact was showing academic gains for participants.
• The funding ultimately was restored as a result of intensive lobbying–in which the legitimate uses of evaluation data for informing social policy development and funding decisions were left undiscussed and hence once again relegated to the margins of social utility.

Here’s my take on the situation:

• Mathematica’s report is clear that this represents only the first year's worth of data and explicitly notes that much more comprehensive and meaningful data will be forthcoming after the second year. It also cautions very explicitly against drawing any conclusions about program effectiveness from these data.
• I believe the disagreements among evaluation professionals would have stayed in-house - as happens all the time - had the Mathematica study not led the Bush administration to propose cutting the 21st Century budget by $400 million. And here Mathematica is not at fault - the report is clear that this represents only the first year's worth of data and explicitly notes that much more comprehensive and meaningful data will be forthcoming after the second year.
• I think the Mathematica study is serious, rigorous, and honest about its early findings, and the Bush administration arguably misused the findings.
• Looking at both the Mathematica 21st Century Learning Center and the PSA TASC evaluations, I would propose the following "bottom line" conclusion – namely, that both find pretty much the same things to be true:
  

• From my perspective, we have, in these two evaluations, the very best research efforts to date in the after-school sector; and, interestingly and not trivially, they show similar findings! And further, it is my hypothesis that these findings will converge as the 21st Century study progresses (unless the issues of 21st Century program implementation that have been documented are indeed serious enough to undercut the achievement of outcomes - an issue of program quality, not evaluation methodology). We will have the opportunity to see if I am correct in this belief.

Ultimately, if one is looking for the best information on what does and doesn't work in any program area–including after-school programming–it is essential to conduct scientifically rigorous evaluations (using both qualitative and quantitative data) to illuminate both how well programs have been implemented and also how effective they are in producing the outcomes that have been targeted for their participants. Both the evaluations that I've discussed are strong and meet standards of scientific rigor. But neither is without its critics. The results of both will consequently be open to ongoing debate. And, as they are for almost all youth programming, in the end the positive outcomes produced by these programs will prove at best to be modest. So, that’s as good as it gets when we use large-scale, multi-site experimental and quasi-experimental program evaluations–a sober but honest assessment of how to learn “what works” in after-school programming using this approach.

With this as my context, for me the essential question becomes “How can–in a day-to-day practical sense–such modest and perhaps even equivocal evaluation research findings be used by those who provide services to young people in the after-school sector?” My answer is that such research must become the empirical basis for program design and implementation.

But even in cases where this is done, it seems to me, the potential for evaluation to add great value to service provision remains unfulfilled. And this is the area I want to discuss briefly in the time remaining to me.

Building Program-Based Evaluation Capacity to Assess Participant Outcomes

Once we have selected an organization for such an investment, we work with its leaders to help them develop a growth-oriented business plan that looks 3-5 years into the future. The first phase of business planning consists of formalizing the organization’s theory of change. This consist, among other things, of:

• clarifying its outcome objectives for those who participate in its programs,
• specifying the group(s) it seeks to serve,
• describing the program elements through which it intends to help participants achieve the targeted outcomes, and
• identifying the human, material, and fiscal resources needed to deliver the services as intended.

This theory of change becomes the core of the grantee’s business plan for growth. It becomes the guide whereby the organization organizes its activities to achieve its strategic goals and objectives. And it provides the framework within which each organization can begin to examine its programming for what works and what doesn't work.

As part of business planning, we look at the kinds of data that grantees routinely collect about participants. Often, we find considerable gaps in what is collected – which typically is insufficient even for the purposes of monitoring, managing, and improving the quality and effectiveness of their programs. In such instances, we work with grantees to include, as part of their business plan, developing an evaluation system as part of building the core capacities that are essential for their success.

We have come to think of the building of grantee evaluation systems in terms of three stages:

1. Participant Profile. Every human service organization must know in detail whom they are serving, and have some basic demographic information about each and every client. At the very least, they should know the name, residence, age, gender, ethnicities, and socioeconomic status of every program participant. For youth-serving agencies, it is also essential to document basic information about the family. And, for organizations wishing to learn about what results or outcomes clients achieve, it is essential that they capture appropriate information about how each new client rates on the key outcome indicators that will be used to measure programmatic success–usually at targeted points in the course of a client’s participation, and also after he or she leaves the program. Implementing this most basic phase of an evaluation system can take several years.
   
2. Participation Patterns. Participation information provides a description of “how much” of a program the participants actually get. These data are essential. For instance, information about program participation patterns can help organizations think about how they engage young people and keep them engaged. Also, participation data can be combined with outcome data to look at whether the participants are getting “enough” services to produce desired outcomes. In fact, it’s probably not very useful to look at outcomes until there’s confidence that participants do receive the level of services intended. Organizations can’t really engage in ongoing program quality management – let alone improvement–without systematically collecting participation data. Implementing this evaluation capacity often takes another two years or so after an organization has implemented its participant profile database.
   
3. Participant Outcomes . Documenting information about the progress and outcomes of clients is the basis for assessing whether the young people participating in a program actually are benefiting from it. Most people think of this evaluation component as answering the question “Does the program work?” However, exploring data about progress and outcomes can be very useful in helping programs think about other questions - such as “Does the program work better for some participants than for others?” or “Are some outcomes easier to achieve than others?” or “Does the way services are provided affect outcomes?” Here again an organization’s evaluation capacity becomes the basis for program management and improvement–as well as the means for looking at how effective its programs really are. Designing and implanting the indicators, measures, and methods through which outcomes are assessed, also can take several years.

The point I am trying to make here is that building an evaluation system is essential for an organization to manage its programs well and to learn sufficiently from its operations to assure the maintenance of high quality and effective programming. Consulting to grantees about the design, implementation, and maintenance of such systems clearly is of great value to those that are committed to serving their clients well. This builds on the theory of change work with an organization and is grounded in a commitment to increasing grantee evaluation capacity in the service of helping each achieve success on its own terms.

It only makes sense to assess program participants’ outcomes to test programmatic effectiveness when an organization has reached sufficient developmental capacity to offer its services:

1. reliably,
2. at a high level of quality,
3. to a significant number (hundreds) of service-recipients,
4. with high levels of participation and low levels of premature drop out,
   
5. over an extended period of time (years),
6. with a full range of evaluation data having been collected, and
7. within a secure and sustainable institutional setting.

At this point, if the organization indeed is serving those whom it intends to serve, and if pre- and post-test data show participants consistently achieving targeted outcomes, it would be fair to assume that the program appears to be working. From what we have seen in our efforts to find high-performing organizations, of all the after-school programs currently operating, few meet these criteria.

In order to reach a substantiated judgement that participants’ outcomes are achieved as a result of their experiences in a program, outcome data of participants must be compared to those achieved by comparable individuals and groups not participating (a quasi-experimental evaluation approach). Very few after-school programs have achieved this level of evidence that they work.

Finally, in order to confirm scientifically that a program is working as intended for participants, its impacts must be established using an experimental research design that includes the use of randomized selection of program participants and a control group. And even when this has been done at one location, it would be unwise to assume that impacts will necessarily be achieved in a like manner at other locations with other staff and other participants. At best, only a handful of after-school programs can claim this level of evidence for their effectiveness.

But I believe in the value of this simple, step-wise approach. There is a terrific payoff to working with organizations–in this case, after-school programs–to help build their evaluation capacities as part of learning about “what works” in after-school programming. If we do so, long before we can be sure about “what works” we can be confident that such organizations will be busy assessing key issues of program quality, making ongoing efforts to improve quality, and consequently will be much more likely to be benefiting the children in their programs than are organizations not so engaged.