Welcome to 'Considerations for Data Collection and Management."
What are some considerations for you to keep in mind as you plan for, collect,
and manage your performance measures data?
This short introductory course is designed to give you a few very simple
and understandable questions and examples to help you develop or hone the plans.
Here are some questions to consider as you start the project.
* Why are you doing the project?
* What change/outcome do you want to create?
* What strategy (or intervention) are you going to use to accomplish the change or outcome?
* Can you measure the change or outcome?
* If you can't measure the outcome, are you sure you are doing the right thing?
The performance measures data must be carefully considered from the very beginning
so that you are sure that your work will result in change you can measure.
If you are just starting your grant year, you're planning what outcomes you want
to accomplish and how you might measure them.
If you are already in your program year, you know what outcome you have committed to measure.
Your grant application was approved
and your volunteers are actively implementing your program's strategy
or intervention in the community.
This is a perfect time to consider data collection and management in more detail.
You can make tweaks for this year and changes for next.
So there are two questions to ask yourself: * Are you clear about WHAT to measure?
AND * Have you researched the most effective way to measure this?
As you're thinking about what to measure, you could enter your term
into an internet search (for example, "confidence in oneself")
and you will find many existing resources to help you identify the options about what
to measure, understand the effectiveness of previously used measures,
and the feasibility of implementing that measure.
As you review these, you should also think about what resources you have available;
what connections, relationships and agreements do you have with your sites
or partners to get the data back?
There is no one best way; do what works for your program.
Your chosen data collection methods should be appropriate to what you're trying to measure;
they should be rigorous - meaning you will get information that is complete
and that you can rely on - and the methods should work for your program.
Here's an example that we can all relate to
and see "Are we really asking the right questions to get to our desired outcome?"
Our example features a driving school because it is easier to see the disconnect
between what you say you are going to measure and what you actually do measure
when it's a subject that you don't work with.
We all come with neutral eyes and can see how it works.
The ABS Driving School has a set course of study including both classroom and on-road lessons.
The desired outcome is that the student achieves a particular skill level
to earn their driver's license.
We need to come up with a way to measure the student's driving skill level.
We will begin with some seemingly straightforward questions
to illustrate how the phrasing of a question can yield answers
that don't reflect the outcome we're looking to measure.
This will let us see where the disconnect happens and what can be done to correct it.
After the course we could ask the student, "Do you like driving?"
And the student might answer, "I love driving!"
That tells us about their enthusiasm, which is great to hear, but it only speaks
to the student's attitude and not their skill level on the road.
While student attitudes toward driving may be interesting or useful information to know,
it doesn't relate to the outcome we are trying to measure and report.
So we change the question and ask the student, "Do you think you are a skilled driver?"
The student answers, "I think I am a GREAT driver."
Framing the question this way, we get back a self-perception,
but we don't know if it's accurate.
It's a subjective view and does not give us an objective sense of the student's skills.
So we change the question again to "Do you know the state driving laws?"
Here is a question that gets an objective answer.
The student either passed the state driver's test or they didn't.
But it's measuring their knowledge, not their skill behind the wheel.
Can they implement what they know?
Since we are trying to measure a skill, we have to ask about a demonstration of that skill.
In this case, 'did you pass your road test'?
The road test is done by a trained tester with a checklist who can measure
if someone is proficient in driving and is eligible for a driver's license.
What we learn from this example is that how you ask the question is really important.
Sometimes you ask a question that on the surface may seem
like it will get you the desired performance measure,
but when you tease it apart you can see it is not quite hitting the mark.
So that is something you need to think about as you look over the questions
in your own data collection instruments: have you asked the right questions
to get directly to your measured outcomes?
Think about what information you really want.
Are you trying to understand how someone feels, their opinion, a subjective answer?
Those may be appropriate in some cases, but if you're going for something objective
that is based on facts and data that are measureable,
you may want some of both kinds of answers.
If so, remember that how you ask the questions is different, which we'll cover now.
As in this example, sometimes there are good reasons to measure both knowledge
of traffic laws and on-road driving proficiency.
It all depends on "What are you trying to measure?"
If I'm the traffic law instructor, I'm trying to measure whether people learned the laws.
If I'm the person in the passenger seat teaching parallel parking, I want to measure proficiency.
This leads us to the next slide on validity.
Are you collecting all the information you need to report your outcome measure?
In this example, your performance measure is the literacy rate
of high school students in grades 9 through 12.
But the information you are collecting is only for one of the four grades, the 10th graders.
So, is this data complete?
Are you collecting all the information needed to report your outcome measure?
No. To be complete, the award recipient would need to collect data
on students from grades 9 through 12.
It's also important to be consistent about how you collect data.
For example: * Do you collect data from all of your sites in the same way?
* Do you train new data collectors on what data they are collecting and how to collect it?
It's important to build in checks and balances
to make sure the data you are collecting is accurate and consistent.
For example: Do you regularly check your data to make sure there are no errors or mistakes?
This will make it easy to catch common errors such as in data entry.
Do you have written processes and procedures that outline how you collect and manage data?
These will make it easier to train new staff and keep your data collection
and management consistent over the lifetime of the project.
To review, we have talked about some questions that you can consider as you start planning for,
collecting, and managing your performance measures data.
This list is not designed to address every question one can consider regarding performance
measures data.
Rather, it is a basic introduction to questions that are helpful to think about.
Thank you for participating in this session on Considerations
for Data Collection and Management.