Before you turn to the more detailed modules
that constitute this tutorial series, we wanted to step back
and offer an overview of the steps and standards and some
of the underlying philosophy behind our framework.
Uh there are two major philosophic points
in our framework.
The first is that planning, performance measurement
and evaluation are all part
of a continuous quality improvement cycle,
and so you'll see throughout the tutorials
that our goal is always to see evaluation as something
that feeds planning and is fed by planning
in performance measurement.
The second point is that our CDC evaluation framework is a is a
type of what's called utilization focused evaluation
or utility focused approach.
What that means is that we're aiming, of course,
to prove something, but more important we're aiming
to improve something, a program, a policy or procedure.
And so you can see from our framework
that that sixth step is ensuring use and sharing lessons learned
and that's the essence of what we're trying to accomplish.
So you see throughout the steps and standards
and throughout these tutorials
that the goal is always what can we do to make choices
that will lead to more use of findings by more people.
Let me quickly walk through the steps one by one
and then I'll talk a little bit
about how the steps are viewed together.
So the first step is engaging stakeholders.
That means deciding who needs to be part of the design
and implementation of the evaluation
for it to make a difference.
Now often you've engaged these people at some detail
when you created the project itself.
So if so, you're piggybacking on that.
Sometimes there are unique stakeholders you're going
to want to engage.
But we'll see that this flavor of stakeholders,
although we call it step one,
flows throughout all the steps of the framework.
The second step is called describing the program.
And here we want you to draw a soup
to nuts picture of the program.
We want you to make sure there's clarity and consensus
on what the activities of the program are
and what the intended outcomes are.
Sometimes we use a logic model for this.
That's certainly not the only way to do it.
The third step is where it all comes together.
We call this utility focused evaluation,
and so this focusing the evaluation step is really the
big leveraging part of our approach.
Here you're going to decide
which evaluation questions are the key ones.
Michael Patton, the famous evaluator,
once said that for any complex program there are 50 viable
evaluation focuses.
So we'll show you using some decision rules how to trim
that down to the most important questions
for this evaluation this time.
Now we call those steps steps one, two, three,
but the reality is you can really start
with any one of those steps.
You just need to hit all three of them before you proceed
on to data collection.
And and also you'll note, and I'm sure you'll experience this
as you go through these tutorials, that insights
at any one of the three steps may cause you to cycle back.
So you may start on step one with stakeholders,
get to step two and describe your program, then suddenly
when you're setting your evaluation focus,
you identify an outcome perhaps you hadn't realized before,
or a stakeholder who turns out to be important.
Alright, in my own work I tend to start with step two,
the program description, and I use that myself
to frame the stakeholder engagement
and then the evaluation focus.
But any of these steps in any order will work.
The important thing of our framework is
that you try all three before you proceed
on to data collection.
The next three steps are really similar
to any data collection procedure you've ever been part of,
but you'll see that we call these titles something a little
bit different than you're used to,
and we'll explain why in a second.
So step four is called gathering credible evidence,
and this is the point at which you'll write indicators
and you'll make your choices
about data collection sources and methods.
In step five you're going to put that all together
and you're going to draw conclusions.
You're going to review and interpret your evidence,
you're going to analyze, actually,
review and interpret your evidence and use it
to determine whether or not the program has been a success
or a failure.
And then finally it all culminates
in use of lessons learned.
If we've done a good job, the whole purpose of this is
to use the evaluation results in a meaningful way.
A key point about our framework is
that those early steps are an underlining guidance for all
of the things that we do in the evaluation.
And particularly this idea of stakeholders and focus end
up uh influencing how we make those choices
at steps four, five and six.
So that's one of the reasons
that we call those steps something a little bit different
than you may be used
to in normal data collection processes.
So for example, we don't call step 4 collect the data,
which is the core task you'll do.
We call it gather credible evidence.
Why? Because if you've done those early steps
of engaging stakeholders
and describing your program correctly,
you already know what the preferences and values are
of those stakeholders, which kind of evidence they,
they uh give credit to,
which kind of evidence they don't give credit to.
You'll find some stakeholders who only want qualitative data,
others who think qualitative data is bologna,
and they don't believe anything if it's not in numbers.
In step five we don't call that analyze the data,
we call it justify conclusions.
Why? Because as we all know, there are data
and then there are interpretations of data,
and at this point the data don't speak for themselves.
They're always seen through a prism of values and preferences
that people are going to bring.
And how will we know that?
Well, we will have known that already from engaging them
as stakeholders and thinking through the evaluation focus.
Then most importantly, and obviously we're not going
to call step six report findings,
we call it use lessons learned.
Why? Because it's a utility focused evaluation and what
that means is we don't do the evaluation
until we have identified at the start who the intended user is.
The payoff for that at step six is that we're not doing a report
and running around wildly to find someone to use it,
rather we're creating a product that meets the needs of a,
of an identified market that we,
that we determined at the outset.
Finally, let me spend a few minutes on the standards.
The standards are at the center of our framework
and while some people see those as a way to do evaluation,
they actually play a much different,
and in some ways, more important role.
The standards help us make informed and good choices
at every individual step.
And what they help us do is to remember
that there's no one thing called a right evaluation
or a right evaluation design.
It's always going to be situation specific.
So rather than telling us how to conduct evaluation,
what the standards do is help us sort through the many,
many options we have in any one of the six steps.
Even a moderately complex program can easily identify
dozens of stakeholders.
Even a moderately complex program has hundreds of ways
of gathering credible evidence.
The standards help us decide how to stay at the right level,
how to keep from under kill,
but just as importantly, from overkill.
There are four standards.
There are actually 30 standards, but they fall into four groups,
and that's what I want to talk about.
The first group of standards relates to utility.
Who needs the information from this evaluation
and what information do they need?
If you apply this standard, you can easily narrow
down 30 stakeholders to the five or six that matter most
for this evaluation this time.
You can narrow down a hundred ways of collecting data
to the six or seven that are really going to matter most
for this evaluation and the intended users
of these findings.
The second group of standards is feasibility.
How much time, money and effort can you really put
into this evaluation this time.
And the feasibility standard is very important.
So many people step away from evaluation
because they bring a very, very high scientific bias
to what an evaluation design should look like.
The feasibility standard reminds us there is no one right design.
Rather there is an intended user of findings
and there is a budget and other constraints,
and then when you meld those things together, you'll figure
out if you've got the time, effort and skills
to do the evaluation you need to do.
The third one is proprietary, or ethics.
Is there some approach we need to take or someone we need
to engage in the evaluation for it to be ethical?
So at evidence gathering we're always going
to ask ourselves are there ethical questions
that would lead us to one data collection method
as opposed to another.
At the stakeholder step we're always going
to ask ourselves is there a stakeholder
out there I absolutely need to engage
to truly understand this program.
If I'm doing a community intervention, for example,
it would be unethical in many ways to not engage as one
of my stakeholders the people participating
in the intervention so I can detect any negative effects
or negative side effects.
And then finally accuracy.
What design will lead to accurate information.
And we put accuracy last for a reason.
Not because it's unimportant, but because in the old days,
accuracy was the king of the standards or the only standard.
And what we brought when we brought the accuracy standard
into prominence was a very scientific bias
or scientific orientation to design,
and there's absolutely nothing wrong with that.
Often it's exactly the most appropriate design to use,
but just as often it's overkill.
And what accuracy does by being our fourth standard is
to be seen as in the service of utility and feasibility.
I'm going to ask myself who needs these findings?
I'm going to ask myself what constraints of skill and time
and effort do I have to bring to this evaluation?
And when I put those things together,
I'm going to then say is there a design
out there that's accurate enough to meet the needs of this user
at this time and still stay within the bounds of my skills,
my time and my budget, right?
And viewed this way, often the answer is yes,
there absolutely is.
Whereas before using only a scientific standard,
we would often end up not evaluating anything
and miss opportunities to really learn more about our program
and how to improve it.
So in short, six steps and four standards help ensure use
of findings, and that's the whole goal of this,
to get some use of findings for program improvement
or conceptual understanding or a host of other uses.
The steps are systematic, but they're also dynamic.
The steps interact with each other,
and they are inevitably iterative,
and that's a good thing.
The early steps are set as a frame of reference
to help us make better choices in the later steps,
and then the standards are at the center of the framework,
and they help us at each step.
They either broaden our view and keep us from under kill,
and or they help us hone in on the best options and keep us
from overkill and remind us
that evaluation is situation specific,
and good choices this time may not be the same choices
next time.