(slide 1) Directions for Expert Review - Establishing Content Validity for COED rubrics
(slide 2) First of all, Thank you for this work!
This work is integral to our accreditation.
We sincerely
appreciate your time and willingness to participate.
We cannot move forward in this effort without the
assistance of our expert panel partners.
Finally, If you have any questions as you complete this task,
please don’t hesitate to ask.
Let’s begin …
(slide 3) We will begin by talking through each phase of the expert panel review process.
The three
phases of this work include: reviewing the materials, completing the tasks, and then
winding up the final
steps by submitting the information back to UNC charlotte.
Finally, while we want your feedback to be
thorough, we don’t anticipate that this process will take great amounts of time.
It’s fairly
straightforward and we’ve tried to make it as painless as possible.
(slide 4) We begin by reviewing the materials sent to you.
These materials may be hard copies or more
likely, electronic versions of document.
These documents should include 1) A letter of purpose,
explaining why you were selected for this work.
This may be in the body of an email or it may be a
separate document; 2) A copy of the assessment instructions provided to candidates; 3) A
copy of the
rubric used to evaluate the assessment; and 4) The response form aligned with the assessment/rubric
for the panel member to rate each item.
While the first three items are informative, the last item, the
response form, is where you will record your responses.
(slide 5) The primary objective of this process is to establish content validity for each
rubric.
As an expert
reviewer, your feedback ratings on the response form help to determine the rubric’s validity.
For
clarification, we are using the term “validity” to determine to what extent the rubric measures
what we
(as the authors of the rubric) claim it does.
Your primary task is to rate each of the items on the rubric in
terms of to what extent the item measures the key construct it purports to measure.
Key construct, or
“overarching idea,” should be identified on your response form, along with an operational
definition of
the key construct, so you understand exactly how UNC charlotte faculty are using the term.
As an
example, we will use a piece from one of our currently used rubrics.
(slide 6) For example, in the College of Education, we use a rubric measuring multiple constructs.
One of
those constructs is CANDIDATE CONTENT KNOWLEDGE.
The operational definition we are using for
CANDIDATE CONTENT KNOWLEDGE is “A candidate’s knowledge about actual subject matter that
is to
be learned or taught.”
Notice this definition changes depending on the grade level/content area.
This is
why reviewers are selected based on their expertise.
(slide 7) On our rubric, there are currently three items on the rubric that are collectively
used to
measure CANDIDATE CONTENT KNOWLEDGE: A: Demonstrates knowledge of content; B: Demonstrates
awareness of literacy instruction across all content areas; and C: Makes content relevant
for all learners.
In considering this example, most reviewers would probably agree that A, demonstrates
knowledge of
content is important in considering the overall construct of CONTENT KNOWLEDGE.
But what about B,
demonstrating awareness of literacy instruction?
While definitely something that all educators should
be knowledgeable about, does “demonstrating awareness of literacy instruction” measure
CANDIDATE
CONTENT KNOWLEDGE, as it is defined here?
Some educators may consider B to be less important in
measuring CANDIDATE CONTENT KNOWLEDGE, depending on their perspective and experience.
This is
why we ask multiple content experts to provide their professional opinions in considering
each of these
key concepts.
Going back to consider item A again, we may consider it to be vague in measuring
CONTENT KNOWLEDGE.
Item A, in essence, simply repeats the key construct.
An expert reviewer may
determine that Item A is too broad and needs clarification to be effective.
This is the kind of feedback
we seek from our expert panel of review.
(slide 8) So what is each reviewer actually rating?
Each item on the response form is rated in terms of
how Representative the item is in measuring the key construct, how important the item
is, and how
Clear the item is.
Each item is rated on a scale of 1-4.
(slide 9) The first thing to assess will be Representativeness.
Using a scale from 1-4, reviewers will rate
the Representativeness of each item in measuring the overarching construct
1 = item is not representative
2 = item needs major revisions to be representative
3 = item needs minor revisions to be representative
4 = item is representative
(slide 10).
Next, using a scale from 1-4, reviewers will rate the Importance of item in measuring the
overarching construct.
1 = item is not necessary to measure the construct
2 = item is provides some information but is not essential to measure the construct
3 = item is useful not but essential to measure the construct
4 = item is essential to measure the construct
(slide 11) Finally, reviewers will rate the Clarity of item
1 = item is not clear
2 = item needs major revisions to be clear
3 = item needs minor revisions to be clear
4 = item is clear
(slide 12) In your materials from UNC charlotte, you should have received either a copy of
your response
form or an electronic link to it.
Following the directions as explained here, please go through and rate
each of the items on the rubric as to how well they measure the identified key constructs.
Remember:
one construct may have multiple items to measure it, or it may only have one item.
Part of your
feedback includes open ended responses: are there enough items to measure the construct?
Too many?
What else would you like to share with us about measuring this item?
Please give your feedback in the
open-ended responses that should be provided on your response form.
(slide 13) One last thing: You’ve been sent copies of the rubric and the original student
assignment in
your packet of materials.
These materials are for you information … you are not assessing the
assignment or the entire rubric – you’re only looking at how well the rubric measures
what it purports to
measure.
Rubric indicators are not part of this … we will look at the rubric indicators when we
assess
inter-rater reliability next year.
(slide 14) When you’re finished, please return the response form to your UNC Charlotte
contact.
If you
have any questions, please contact us.
And once again, THANK YOU for your time and willingness to
assist us in this important accreditation work.