Analytics can change a culture.
I've seen it. And it takes a lot of
things to occur and I know you're all a
part of this and what you want to do
with analytics is try and change the
culture. I will give you my humble
opinion as far as how it works and
what's the secret sauce. And you'll see
this weaved into what's gone on.
Number one: you need data. Obviously you
can't do a whole lot if you don't have
any data. Don't short yourself. Don't get
just the data you need to fill out this
report. Grab everything you can
because someday there will be questions
that can only be answered by data that
you captured previously. Number two: you
need to be able to report it. You need to
have some people who know how to report
it. If you don't have this skill, you can
crowdsource it. We actually did this
within our system as well. Other people
within your system will have other
skills and you can leverage them.
Number three: you actually need researchers.
Sometimes it's more than just reporting
and putting out a report that's really
cool. You need a researcher to actually dig into
your data and look at some hypotheses
and write some white papers and reports.
They can be very useful. And finally,
you can't underestimate this one: you need
relationship capital. You can be in the
middle of the organization doing
reporting and have all these other
things, but if you have no ability to
push it up or down or left or right you
don't have the ability to change your
culture. Finally, this one's a good one:
this really helps you need a good fiscal
or political crisis. There are
moments in time inside the educational
domain who really likes to remain in
1850s mode for the most part they don't
like change but occasionally you get
wrinkles in time where you can push them
and drive them. It usually is a fiscal or
political crisis that they're going
through. Be ready when that happens.
In my humble opinion, in my own personal
timeline, the dawn and driver of
analytics in higher education started
with the student right to know act.
In 1995. So the feds said "campuses,
you need to disclose your graduation
rates to the public." Before this,
nobody had to do this and this was
kind of shocking at the time, but that
act created the graduation rate survey.
Public disclosure of graduation rates.
First-time full-time degree seeking
students track to one hundred and fifty
percent normal time to completion.
This was the methodology we were stuck
with. I'm sitting out here in a community
college system going, "what the heck is
this?!" This caused seething outrage.
Number one, if you're in a two-year
Community College System your first
issue is: "I don't even know who's degree
seeking!" Full-time is not a permanent
status for a community college student
for the most part. If they're
full-time in fall -- they took 12 units --
well, half of them have dropped, half of
them by week three anyway. They
also counted transfer in a rather odd
manner. One to the feds transfer was not
really an outcome. Transfer was
'leakage,' so they commingled transfer
-- upward transfer, transfer from a two-year
institution to a four-year institution,
which is a good thing -- they co-mingled that
with lateral transfer. And ultimately
what we got were community colleges that
had really bad graduation rates, and this
was one of the first drivers for change
as far as, "this can't be the public
disclosure of what we're going to put
out there as our accountability metric?"
We needed to paint a rosier picture.
Because one of the problems with putting
out a really low graduation rate in the
public domain is that basically you're
telling a group of students who
already are kind of marginal, "you don't
have a chance of getting through here."
And it's not a message you want to tell
them. You don't want them believing that.
You want to tell them this is a place
where you can succeed. We needed data.
We needed a better methodology.
We needed to get really creative with defining
success. That's code for being really
inclusive in the numerator and being
really restrictive
in the denominator. This was the game at
the time. Our system up till that point
was volume-based -- annual volume of
transfers, annual volume of graduates.
And as you know this kind of rides
and goes down with the tides as far as
how much money you have in your system.
We had to change this to a rate-based
system because that's what the feds were
doing. It was a rate.
So we convened a bunch of stakeholders --
mostly institutional researchers
in our system to come up
with the ultimate rate for community
colleges. It was called the student
progress and achievement rate or the
SPAR rate. And here's how it went:
First thing we did was we got rid of why are
you here and converted that into "I'm
going to look at your enrollment
patterns to determine why you're here.
Asking a student "why are you here?" is
actually not a very good predictor of
why are you here. So we started looking
at the patterns of enrollment behavior.
And the pattern that we ultimately came
up with is: "alright, you come to our
system and if you take math or English
we figure you're here to get a degree or
transfer because that means you had to
go through the metric process take an
assessment test and get into math and
English which in general nobody does for
fun." Then the numerator was any good
outcome that comes out of this place: you
got a degree, you got a certificate, you
transfer to a four-year institution, or
you became eligible for transfer to CSU
became the combined numerator. This was
the SPAR rate. Lo and behold we got rates
in like fifty percent range!
This was cool and we went to the legislature
and said look here we're not a disaster!
We put it out in the public domain and
everyone looked at us and nobody
believed it. Even though we put out these
rates that nobody believed, first thing
everybody wanted was, "give me the data
set that you used to get that because
you've got all this data." We said,
"alright, we'll put it out there you can
download it, campuses. And we'll give you a
copy, foundation researchers who want to
look at this as well."
And all the campuses took this data and
they built their own local data
warehouses with it. You know, mostly SAS
and SPSS and stuff like this. But
suddenly 113 researchers at 113 campuses
had access to the same data set. And lo
and behold, everybody started talking.
The listserv at the time started exploding
because they're all talking the
same data set in the same language. They
all started sharing their queries. They
crowdsourced each other for answers in
research. Then they came back to us at
the chancellor's office and said "hey,
we're all asking the same question. Can
you just standardized this and put it up
so we don't have to recreate this over
and over and over?" And we said "okay, we're
going to build you the data mart. This
was our first public reporting of a
whole bunch of metrics. We didn't come up
with the metrics. The campuses told us what
metrics they wanted to see. This is the
data mart as it is today in the
community college system. So all the
things you can get public. It's
old tech, but all of the data is out
there and it's a very very useful thing
and suddenly people had all of these
things lifted off of their desks that
they no longer had to do they know where
they can get it. And then came the crash.
Our head count on our system went from
2.9 million to 2.1 million over the
course of two years. We lost more
students than CSU and UC have students.
We had to cut twenty percent of our
course sections. And then came the bill.
The bill sought to improve our system
and also ration money. And what it said
was, "you guys get paid for butts in seats
at first census. We're going to change
this because we don't have enough money
to pay for that. We're going to change it
to butts in seats at end to term."
This is not a good thing. This was a very
blunt instrument. And we were like ...
we're going to fight this one. But we
couldn't get rid of the bill.
Higher ed committees were still like,
"yeah, what's wrong with this? it'll
motivate you to be really good."
And we went "oh god, we're going to have to
own this one, and we're going to have to
cut a deal. alright. No bill. We will own
our problems. Give us
a year and we will fix ourselves." So the
Student Success Task Force was created
to deal with this and we had to invite
in the Department of Finance and the
legislature and the ledge analysts to
sit in on our dirty laundry. And they had
input. They said "address your issues or
face the wrath. You have low completion
rates. You have very poor success in
remedial courses. Transfer, you have a lot
of students who are going all over the
place or not going anywhere or
accumulating 110 units and not going
anywhere. And then you have CTE issues to
deal with as well. Mr. Perry, we don't
like your rate. We're not believing it.
It's too exclusive. Count everybody." So
suddenly we needed to figure out other
ways to measure success and we needed to
do it for a multitude of populations now
because we have to count everybody. The next
thing we did was we hired a researcher.
And I said, "Peter, I need you to look at
the entire student population and we
need to cluster them. We need to figure
out what kind of buckets they're in and
give us you know where they're going and
what they're doing well in the system." So
he does a cluster study. He
looks at the populations of students and
their properties. This is actually a
pretty landmark study for our system.
He takes a group of students -- a first-time
freshmen cohort 165,921 students -- they
accounted for 223,500 FTE over their
entire academic life. He does a cluster
model with them and here are
the clusters that they pop into. You've
got students who are in transfer
clusters, right? You can see what
percentage of the students they are what
percentage of the FTEs they consume. What
are their median
outcomes. So this is a pretty
interesting study and we haven't had
this before. It shows us here's where
we're spending our money, here the
outcomes of this. We actually started
looking at it, in a certain sense,
not by headcount but by the FTE consumed
and whether this FTE is being converted
into an outcome. But suddenly
we had a study to work off of. And we had
the groups that we needed to work on. And
then we began building our next
accountability system. More analytics.
One, we're going to modify our SPAR rate and
say it's not just degree-seeking math or
English we're going to use to get into
this cohort. It's any math or English. And now
we're going to include remedial students
because they're degree seeking too. Our
rates naturally went down, but what we
did was we changed the spin. No longer
is this about "we have poor grad rates."
This is about "remedial is a really big
problem here." Accountability 2.0
went like this: it's not about full-time
or part-time. It's about whether you came
to us needing remediation or not. If you
come to a community college and you
don't have to take any remediation, your
grad rate's seventy percent. If you need
any kind of remediation in math or
English only, it's forty percent.
Three-quarters of them need remediation.
But this changed the focus and
suddenly the legislature was like "Ah, okay.
This is helpful. And in fact we'll help you
pay for this." They gave us money. they said
"fix your remedial problem." They gave us
money. Here's the actual numbers: you go
to Community College in California you
got a one in three chance of getting
through remedial math and a little less
than one in two chance of making it
through English. This is the problem.
Finally somebody got smart and did some
analytics on this cause and effect. And
that was in, our system, Long Beach City
College in 2011. The first thing they did?
They got some data. Partnered with Long
Beach Unified School District. Got the
entire K-12 transcript database moved
over to the community college database.
They started banging on it and said, "how
can we place these students better?"
They looked at the K-12 enrollment factors
that predicted success in their math and
English courses. Here's what they came up
with: standardized tests were weak
predictors of performance. The best
predictors was their performance in high
school courses including, most strongly,
their consistent performance on a wide
variety of assessment methods across
multiple disciplines also known as their
high school GPA. This is where multiple
measures placement in California was
born. Long Beach City College then said,
"alright, there's three ways into
transferable math or English: you can
take the test, you can do it through the
high school exit exam score, or you can
have a GPA of 2.8 or above -- which the
faculty just went wiggy over. But guess
what! The number of students in placed in
transferable level math or English did
this: went from nine percent thirty-one
percent math, thirteen percent to
fifty-nine percent in English. And the
faculty went, "Oh yeah, but come on. The
course success rates are going to bomb.
You can't shove these people in these
courses. Actually, the course success
rates were pretty close to the rest of
the general population. And if you shove
that many more people and they're still
getting the same course success rate, you
successfully got rid of remediation. Move
on to transfer. Transfer's complicated.
First thing we had was quantifying
transfer of success. Student right
to know counts it as a subordinate outcome.
You count your degree seekers first.
Transfers come later. Lateral transfer is
commingled. Getting transfer data at
the time was not easy. We cut ourselves
the very first system-wide deal with a
National Student Clearinghouse in the
late 90s. Because what they wanted were
our campuses feeding them data. They were
in rapid expansion mode and they had about
ten community colleges in their system
at the time. We gave them all 113 in
exchange for "I get to match unlimited
with you." We also matched with CSU and UC
directly. So we had this perfect
accounting of transfers. We created
what's called the transfer bucket. Don't
ever limit your matches. We got
transfer data from before, during, and
after
their time in Community College. No
limits on them. And because of this the
amount of analysis we could do exploded.
First thing we said was, "hey the
graduation rate survey says they have to
be first-time students but I got all
these enrollments before they got here
so I'm going to kick them out of the
cohort because I'm doing this match. Hey,
says first time. Obviously they went to
college in 1978 and came back to
community college in 1996. Not first time.
Out of the cohort. They had lower success
rates anyway. Okay, we could measure swirl
perfectly and we could report swirl in
the GRS perfectly. And they hated us for
it. I got phone calls "you got transfer
rates of 75 percent. How can that be?" and
I said "Fullerton and Cypress are next door to
each other and they share their programs
with each other. You allowed it to happen, Feds.
Change your methodology. And until
then, I'm reporting seventy-five percent.
It's your methodology. But what we really
learned that was kind of ground changing
was what was happening to transfers in
other sectors besides public in
California. And primarily where our
students were going to for-profit
institutions. We didn't have this before
the Student Clearinghouse match. Take a
look at this 10-year examination of
students going to in-state privates --
which is actually a small amount -- and
for-profits which is a big chunk of that
over the course of ten years. Eleven
thousand. Eighteen thousand ten years
later. This far outstripped transfer
to CSU and UC. Quite a few going
to out-of-state too. Most of them are
going to the University of Phoenix.
Most of them were underrepresented. Most
of them were older students. Most left
without doing a whole lot in the
community college system. They weren't
traditional transfer. They weren't
getting their degree. They weren't taking
math or English. And when we asked the
campuses "what's going on?" and they said
"yeah, they're parked on the quad. They're
parked in the hallway of the business
department. They're parked in the cafeteria.
They're sitting there in their booth,
grabbing them as they come by, saying
'just sign here and we'll get you out of
this place where you can't get a course.'"
This is why the University of
Phoenix was cleaning our clock. CSU and
UC? They showed up once a month. "Hi, we're
here." Phoenix was there all the time. What
happened shortly thereafter ended up
clearing up the problem. Number one: Phoenix
and other institutions were nabbed by
the Feds for some predatory practices.
And suddenly their presence
diminished. But the good thing that
really happened as a result of seeing
this analytic for the first time was that
the legislature got in an uproar and
passed a bill. SB 1440 student transfer
Reform Act created an associate degree
for transfer. And here's what
happened since that day: it's slowly
declined. Final thoughts: analytics is not
reporting. If you're in the
business of just cutting reports, you're
not going to change the culture.
You can put it out there, but that doesn't mean
that something's going to happen.
Remember the secret sauce. You got to
have data. Then you gotta have reporting. You've got
to have research. You've got to have the
relationships to push it. Have it all
ready to go when the next fiscal crisis
looms because crisis is a terrible
terrible thing to waste in higher
education. Thank you very much! have a
great conference.
you