[ Music ]
[ Applause ]
Yes and I am Janet White.
I am an Education Program Director with the Office of Personnel Management, Center for
Leadership Development and I actually work out of the Eastern Management Development
Center.
We were currently, we were located in Washington or Shepherdstown, West Virginia but we have
transitioned into D.C. and we will kind of be splitting our time between D.C. and Charlottesville
because we will be offering some of our programs at our FEI center or institute there in Charlottesville.
I was asked to come because I actually have a bronze-level certificate in applying the
Kirkpatrick Model for Program Evaluation and as indicated I am finishing my doctorate,
yeah, in human and organizational learning in that dissertation phase which let me tell
you, that's a killer.
How many of you guys are familiar with OPM's management and executive leadership training?
Any of you been to the eastern or western center or maybe even FEI?
I appreciate that very much.
We are a revolving fund entity.
We don't get appropriations so all of the pennies help believe me.
We have been providing training for nearly 60 years now.
It's hard to believe but yes, we've been providing leadership and management and executive training
to federal, state, local and foreign governments for nearly 60 years so you could say that
we're kind of the go-to folks when it comes to employee development and training.
And I was asked to come here today because, to talk to you about training effectiveness
versus effective training.
Now, based on our own experience and we've had a lot of it, we truly understand the difference
between those two and we've suffered through some arduous times in, with our training programs
because of it.
Our own experience shows us that it's not enough just to have effective training programs
but you need to determine the training effectiveness as a whole because it affects your organization.
So, you guys are probably familiar with this, okay.
Well, in preparing for my presentation I was asked, I wanted to get sort of an idea of
what you guys have done what kind of training you have offered before in the past and just
sort of to get some insight into your particular training curriculum.
I was told that you guys put a lot of effort and time into putting together this training
curriculum and I could tell, I was very impressed when I saw this because I could see that you
had a very clear delineation of your learning levels as well as your career paths so I applaud
you for that effort.
It was a great effort.
But my question to you today really is, is this training effective?
And if so, how do you know that?
Okay, well, let me, let me talk about a little bit, tell you a story about our organization.
As I mentioned earlier we've been doing this stuff for nearly 60 years.
But back in 2006 OPM, my part of the organization went through a very lengthy and arduous OMB
Performance Assessment Rating Review.
Oh crap is what we said, truly.
But we were like, no problem.
We've got all this experience under our belt, no problem.
We got all these great ratings from our participants.
We've all this stuff.
We had just gone through an ACE review, the American Council on Education.
They've gone through all of our documents.
We have all the documentation for our courses and all that other great stuff.
No problem, we won't have any problem passing this thing, boom.
Well, OMB looked at all the documentation, everything that we presented, all of our language
of governing what we did and why we did it, everything.
The report came back, results not demonstrated.
Oh my God, we were devastated, absolutely devastated, results not demonstrated.
How could that be?
Well, what it really meant to us was the fact that we needed to do a better job in how we
measured and evaluated our training programs.
It didn't mean we were doing a bad job, it's we weren't doing the right thing.
So, again we needed to look at what was the effect that we were having on the federal
government as a whole?
Would the training we were providing enable participants to go back to their jobs and
do something better?
You know, were they in effect better leaders, better supervisors, and better managers within
their own organizations?
How did we know that our participants were doing better back on the job?
What measures were we using to capture those particular results?
You know and were we using those same measures in all of our classes?
Well, we didn't.
We did not measure our classes the same way.
We did not use the same evaluation methodology in every single one of our classes regardless
of what level it was.
We could not definitively demonstrate training effectiveness so we said okay this is a really
hard lesson for us to learn.
What do we do now?
Well, we had to relook our entire evaluation process and methodology on terms of how we
gathered data, when we gathered the data, why we were gathering the data, did it make
sense to gather that type of data and what was the real purpose of that particular data?
You know we looked at all the questions that we asked the participants.
We also said, are we asking questions of their supervisors and if not, why not?
You know, we had to really take a very long, hard look at how we could better demonstrate
training effectiveness.
So, by the end of the session hopefully you'll gain a little bit but greater appreciation
and understanding of the importance of training effectiveness.
Hopefully you'll walk away with some ideas of how you can improve the ways you guys measure
the effectiveness of your training programs because you've put a lot of hard work into
it, there's no doubt about it.
And also I hope that you look at your evaluation results and you learn and understand how you
can use that specific data to inform some of the decisions that you make around your
curriculum changes as well as your resource levels, what you dedicate to programs and
what you don't.
You should be able to use that evaluation data to support and improve your strategic
planning initiatives and actually lead to greater organizational effectiveness.
So, what's the difference between effective training and training effectiveness?
Well, for one thing training effectiveness, the focus of the assessment is on the training
program.
Are you looking to make improvements to the content?
Is there a better case study that you could use to illustrate that particular concept?
Should we add another exercise to allow the participants to practise a little bit more?
Do we need to change the instructor because they really were engaging, I mean they were
really [inaudible] so you know that's effective training, again focus on the content.
On the other hand, there's training effectiveness.
Well what does that mean?
Well, the focus of the assessment is on the impact the training is having on the organization
as a whole.
How well did the training meet the developmental needs of those participants?
How well did the participants master the training and that content?
How well did the participants transfer that learning back on the job?
How well did the training contribute to the achievement of those organizational goals
and mission?
That's the difference between effective training and training effectiveness and as we go through
this you'll see a little bit more.
So, why do we want to measure it?
Well, first well-instructed measure-systems can help determine where there are problems
not only within the organization but also within your curriculum.
Are there employee skill gaps?
If so, what are they?
Does your training program address those skill gaps?
Are there gaps in the performance of the organization that are not being addressed in those current
training programs?
How do you know this?
What are the indicators that point to the gaps in the training effectiveness?
What are they?
Second, the ability to demonstrate real and significant benefits at the employee level
can help your organizations justify the need for more resources.
And it's not only the employee level but it's also at your organization level as well.
Most of the time programs fail to deliver based on expectations.
There's usually a mismatch between the learning objectives of the class and the expectations
of the participants.
Agreed?
Okay, there's usually a mismatch between the expectations of the participants taking the
training programs and their supervisors sending them to the training programs.
There's usually a mismatch between the expectations of the HR and training officials and the organizational
decision-makers and stakeholders.
There is also a mismatch that is usually due to insufficient planning and coordination
in the early stages of your training program in that design plan.
Oops.