[ Music ]
[ Applause ]
Understanding and using evaluation results.
Well here's some examples.
These are just two of how different agencies evaluate their training effectiveness and
their results.
The first one is with regard to the evaluation methods used at the Department of Commerce.
They use the Kilpatrick Approach for their program evaluation, and it is considered their
gold standard.
They use a six-point Likert scale in their survey, and they have three variations of
like, and three variations of dislike, and that prevents the survey taker from actually
taking a neutral position, and it doesn't--they don't have an out this time.
They have to provide some sort of an opinion on how that training program was for them.
So they have to provide that opinion.
They--all their courses conduct a level one, the smile sheet, and yeah there are some studies
that say that that you know really you hope that's pretty good and it's more substantial
than people give it credit for.
But commerce says no, I don't think so.
They continue to evaluate their leadership development programs by surveying their participants
and their supervisors three to six months after each program has been completed.
This allows them to get to that level three.
The other thing that they do is alternating years, they actually conduct an extensive
triangulation of three types of data.
They collect qualitative and quantitative data so that one can validate the other, and
they do random interviews, they do randomly selected focus groups, they survey the program
participants, their first line supervisors, and their second level supervisors.
And that data that they collect enables them to determine the organizational impact of
that training program on the operational side of things.
The information is also used to make program improvements not only in the curriculum but
in the organization itself.
The other one is a much larger program, and that was the IRS program; it was their Large
Business and International division LBNI program.
And it was with their new hire training design.
Their mission obviously is to provide top quality service in helping customers understand
and meet their tax responsibilities as well as making sure that they are applying tax
laws in a fair and integral way.
They want to make sure that it's done evenly across the board.
Because the LBNI operational goals is to balance customer satisfaction, employee satisfaction,
and business results, the training effectiveness measures had to be the same.
If not the fundamental objectives of the training would likely skew what their operational goals
were.
So, what they did is they actually redesigned the new hire training program with an eye
towards the business operational goals and their underlying objectives.
And this was to make sure that they were again, it was a balance.
What they ended up doing is under the new hiring training program it said, okay.
Their first original--their training program was okay new hires will get, or they're going
to attend this 40 hour training class.
Okay then they're going to attend you know a couple weeks later they'll attend this 26
hour.
Oh and then there's going to be this rotation, and oh there's going to be this.
Well, okay, but they really couldn't tell and they weren't collecting data of whether
or not it was being effect--if that training was effective or not.
So what they decided to do is redesign this.
It says upon completion of the new hire training the employee would be able to perform a quality
audit in accordance with a strategic focus.
And so now at the end of their training program, they have to come back and actually do a quality
exam.
They have--they actually do an exam, and they conduct this quality audit.
And they have several--seven, excuse me, enabling objectives that link back to the business
goals, and one has to do with their initial planning of the audit, the actual audit plan
itself, the inspection and their fact finding, processes and procedures, the actual application
of the tax law in its determination.
There is a mid-cycle analysis and proposal and/or resolution submission.
There is also an analysis of the work reports and papers themselves.
And then they also have one administrative objective which has to do with the actual
procedures on how they do all of that.
So all of that now is linked to what their operational and strategic goals are so they
can see with through this training how is it impacting the end result.
They also--they actually established a quality measure system so that as they do these audits
they can actually go back and take a look at whether or not the exam itself was fair
and was accurate, and that it was you know--that the measures were actually being captured
correctly and whether the questions were right, and whether those were the right questions.
So they would do a quality check even on the exam process itself.
And then they would actually provide those statistics to all of the different levels
of organization.
As a result of the quality review, guidance is given to management and technical persons
for specific steps to take for remedy and improvements in their process and the exam
itself for all these new hires.
So I mean they went through a very lengthy lengthy process to redesign their new hiring
training program itself; not just to hear some training programs.
Now, how does that link to what we really want to have done, and what should it look
like?
So back to this, again.
You guys worked really really hard in developing this training curriculum, but let's take an
example of some of the training effectiveness data that you collect for your courses.
I asked for this data as I was preparing for this, because I wanted to see what you guys
do you know so I can you know maybe point out some things that you might be able to
do differently.
Again, great framework, but how do you know if your training is effective?
So this was one of the first examples that I saw.
Can you determine training effectiveness from these types of questions?
And I apologize for these are a little bit hard to see when you try to put them on slides,
but again, is was the appropriate amount of time given to the topic?
Was there ample opportunity given to ask questions?
More going towards effective training versus training effectiveness?
What about this one; can you determine training effectiveness from this type of survey?
What are you looking for?
Okay, yeah you're looking for feedback on your speakers but what are you--what do you
want to know?
Were they good?
What module was it?
Okay, satisfied not satisfied?
Oh, and there's also one in the middle kind of neutral.
Everybody's neutral.
Everybody's always good.
Okay, how about this one?
I like this one because you're starting to what extent.
Questions posed you know and that's the type of questioning you want to do using the Kilpatrick
type of model.
But I think you could improve on this because you want to be able to provide participants
a place to explain their answers; open-ended questions, explain why they answered it the
way they did.
This one, again, this is the last one, and I understand this is the more recent stuff.
And again, this is getting even better.
It'll help you get to determining training effectiveness; however it falls just a little
bit short, because again, you need to allow the participants to elaborate on their responses.
The qualitative data in collecting that you collect is sometimes more important than this
little qualitative part.
Excuse me, the qualitative data is more important than the quantitative data, because that's
where you really get a lot of your meat from how that training was really received.
Here's a quote that I found from Albert Einstein that I really think clearly represents the
dilemma that most training organizations face when they're trying to collect data to help
them try to demonstrate it.
"Everything that can be counted does not necessarily count, and everything that counts cannot necessarily
be counted."
It's what we've been talking about.
It's a hard thing to do, but it can be done.
So what does the FDA training evaluation data really show?
Well, here's what I concluded when I looked at your information.
Again, the first example you can use it to make improvements in that particular training
program; didn't quite collect enough information about the training program itself.
In the second one the questions were kind of limited to the course objectives and the
amount of time and so forth.
Again, it's more about the content and not the experience itself.
Using your five-point Likert scale, again, creates that bias.
It allows people to either be really neutral or really favorable, especially when they
want to get out of here at the end of the day going oh good God is she ever going to
shut up type of thing.
You know to be able to make sure that they can really provide you with a response that
you can--a good response that you can use.
And it appears that again, those were four different types of surveys, but and it appeared
that you were using different types of surveys for different classes.
You may want to think about making sure that you use the same survey for all classes at
all levels, because what that will do is it will allow you to create that important baseline
that as you collect data over time you can see the improvements over and over and over
again.
Again, this is kind of what you should do; just my recommendations.
You want to have questions that's going to elicit more information from your participants
to really help you determine whether or not that training is effective.
Questions should be structured so that they gather data about the level of the knowledge,
skills, and abilities before and after the class.
And you can do that even in an after class survey.
He's like okay.
Coming into this class you know how much did you think you know, and let them elaborate
on it.
Now, after the class what were you--what's different in your knowledge?
Questions should be open-ended, and again, giving them space to allow them to elaborate
as much as possible.
Again, you can kind of--you don't want them to be long winded and that's okay, but I mean
give them a decent amount of space to be able to react to it.
While most evaluating--most evaluations use odd number of Likert scale answers, again,
you may want to think about moving to that even number.
Again, like the commerce does they use the three like and three dislike, because it really
helps you to kind of get to that you know an answer and nobody can really take a neutral
position.
And as I said earlier, standardize your course evaluation across all of your courses and
across all your levels, because the more consistent you are over time that really will help demonstrate
your organizational effectiveness as well with your training programs.