- Hello everyone, thank you for staying
for this last paper of the session.
I know it's after lunch and everybody's a little bit sleepy,
but today I'm going to be sharing
some of our long-term badging work.
I'm Caroline, and my co-authors are Adam Bell,
Edgar Onofre, and Katie Davis,
and we're from the University of Washington
Information School.
And I would like to start out with a question.
How many of you have received an email saying,
"Oh, we're going to be switching to a new payroll system."
Or, "We're going to be switching
"our content, our course management software
"to a new system."
And just you get this email and you have no choice,
and you have to switch, and it's a new system
that you have to use, and how does that make you feel?
Does that feel great, or does that feel like,
"Oh no, now I have to change everything
"to this new system."
Or maybe some of you have sent those emails
or created that new system.
So have we, we've spent the past several years
working on a new digital badging system
for an ongoing program.
So this program at the Pacific Science Center
called the Discovery Corps has been ongoing,
a youth science interpretation program, for several years.
So we just in the past several years
implemented a digital badging program.
And I'm sure many of you are familiar
with what a digital badge is.
But for those of you who are not,
even though it was mentioned in Sebastian's presentation
a couple of presentations ago,
digital badges are generally little icons
with some embedded metadata.
In this case, we're specifically talking about them
for learning, so given in after-school programs,
or different digital learning programs,
to provide a way of tracking learning pathways
and demonstrating learning achievements across settings.
So in our particular case, we're looking at,
"How do we design badge systems
"that align with the needs and values
"of the stakeholder groups
"that we're particularly working with?"
And in particular, we want to look at,
"How do we develop badge systems for long-term use?
"How do we implement the badge system
"and design it in such a way that it's going to be used
"over a long period of time."
That when we implement it and leave,
they don't just go back to what they were doing before.
They don't say,
"Oh, this system is no longer being supported,
"we wanna go back to the way we were doing things earlier."
So we want to particularly focus
on this longer-term implementation,
and how to really incorporate the digital badge system
into the day-to-day operations
and the workflow of the system.
So the Pacific Science Center is a science center in Seattle
with whom we have a research practice partnership,
and in particular, we're working with
the Pacific Science Center Discovery Corps,
which is a group of youth science interpreters
who work at the Science Center
and engage in both work activities and learning activities.
So they develop a variety of science skills and so on,
and we work directly with them
in a participatory design setting.
And so, in particular, we want to look at
how this work can be used to enhance
implementation in the program.
So our first research question was,
"Over the past several years,
"what challenges did stakeholders face
"during the implementation of the digital badge system?"
Question two, "How can implementation challenges
"that we surfaced earlier be anticipated
"and incorporated into the initial design process
"and the ongoing design?"
And, "What role can this ongoing co-design process
"play during the long-term implementation
"of a badge system?"
So in order to do this, we wanted to look at
our large purpose of qualitative data
that we've collected over the past several years.
So in order to do that, we want to talk about
the project timeline.
It actually started before this with pilot work,
but the first official year of the program
was in 2015-2016.
So in that first year,
there were the initial design sessions,
and the initial surveys and interviews with the students,
as well as the initial interviews
with a group of community stakeholders.
College admissions officers, local tech company
Human Resources representatives,
to discuss the potential use in job applications and so on.
And then in 2016-2017 was the badge system launch
and iterative design and testing.
Incorporation of new features,
discussions with the students for their design sessions.
In 2017-2018, feature updates, follow-up surveys,
and interviews and so on.
And now we are currently in 2018-2019,
where we've incorporated some of the things we've learned
earlier and are engaging in long-term case studies,
trainings, and other activities.
So the data sources that we're using
in this particular study
are several of these sources from these different years,
including the various interviews, the surveys,
some of the analytics from the website.
All of the session memos, meeting notes, design artifacts.
So it's quite a lot of different materials.
So it's quite a lot to analyze.
But how we analyzed was through an iterative,
qualitative coding process.
So we built on previous work
in terms of a qualitative coding scheme
that was created through repeated revisions
and triangulation with additional data sources
such as our meeting notes and so on.
And then, over time,
discussing with the different project team members.
And the different coders refined the code book,
and then proceeded to the final key challenges
that we discovered.
So moving on to the findings.
We found three main categories
of badge implementation challenges.
The three main challenges that emerged were
sociotechnical, which focused on the badge system
within the day-to-day operations of the program.
Because when you're inserting a new kind of technology,
a new sociotechnical system, into an existing program
that already has very tight schedule, lots of rules,
lots of things going on.
When you add something new into a program,
they have to find ways to incorporate it,
so there are a variety of challenges there.
There are also sociocultural challenges,
which is when you ask someone,
"What are these digital badges for?
"Why do they have them?"
How it impacts experiences outside the program,
how they share it with others, and that sort of thing.
And then the technical challenges,
which everyone in this room has faced technical challenges.
You don't wanna admit that your system
will occasionally mess up or fail,
but it's an important consideration
because it really affects how your stakeholders
and how the people using the system
will interact with the system.
If there's an initial problem,
the first time they log on, they can't log on,
they're going to have a certain opinion
of how the system works that continues.
Even if it works every other time after that,
if the first time they use it it messes up,
they're not going to be happy.
So those are the three main categories
of challenges we surfaced.
I'm going to give a couple of examples.
I don't have time to dive into
each of our subsections here.
They're in the paper, but we're gonna provide
two little cases here.
The first example I want to provide
is the sociotechnical challenge of access.
In terms of actually logging into the badge system
and using it, the students have a very tight schedule
that they go through at the Science Center,
and often they may not have time
to access the badge system.
They may not think about accessing the badge system
because they're quite busy.
So when we talk to them, we ask them,
"When is a good time?"
So we had a design session
where they mapped out their schedule,
they helped brainstorm different times, different tasks.
They were like, "Okay, we can take 10 minutes
"to build a portfolio.
"We can take 15 minutes to check these badges.
"We can do this."
So we worked very closely with them
to actually design scheduled time within the badge system
to, not within the badge system, within the schedule,
to provide time for them to access the badge system.
They actually have something in their schedule
which is called "HammerTime."
As you can see here, this is actually the HammerTime board,
and it's all these short activities
that they can do during their scheduled HammerTime.
Stop, HammerTime!
Yes, that's the joke.
So they have a choice of these various activities
on the board, and since we had that added to the board,
now they can have time to access the badges
and remember to access them regularly.
So the strategy was to therefore incorporate
the badge system into the schedule.
Make it an enmeshed part of their day-to-day activities,
which seems fairly straightforward,
but until we did the explicitly,
until we made sure that it was there,
we talked to the students regularly
and sometimes they weren't checking their badges.
Our second example is another sociotechnical challenge,
this time focusing on awareness.
And it also incorporates a couple
of the sociocultural challenges,
which I'll get into in a bit.
But the challenge was that because this project
has been running for several years,
the first couple of years, we had students
who were helping us in the design team,
we had our co-designers.
And they would go and talk about the badges,
they would go and talk about the design process.
They were very cognizant of all
the different aspects of it,
and they were very familiar with it,
and they would share it with their peers.
Unfortunately, these are high school students
and they eventually graduate and go off to college,
and then they are no longer in the program.
They're no longer talking about the badge system.
So now that it's in the stage
where we're still doing some design
but we're not doing the intensive levels
that we were previously,
there's less awareness of the badge system.
So we incorporated it into their trainings.
They already have these trainings,
this is not us imposing another training
on some high school students who are already very busy.
This is part of their existing trainings.
And we actually had them train each other.
So some of the more senior students in the program
who have a lot more experience
and who want to work on their public speaking
and teaching skills get taught how to do a training
for the digital badge system,
and then they train the younger students
who are just introduced to the program,
and they teach them about the badge system.
And we also give a little badge
for completing the badge training,
which is this badge up here.
Additionally, they have a tradition in their program
where they have lanyards, where they have pins on them,
so we made pins of this badge
because it was part of their tradition
to have these pins.
So now they have a pin to remind them of the badge system,
and also to be part of their traditions
because it's very important to build this rapport
with the community.
So this not only contributes to the awareness,
but it also helps them understand the value and relevance
because it's explained to them by their peers
who are also in the program.
So there are a lot more of these discussions
and examples in the paper,
but I would like to talk about
a few of our main recommendations
for long-term implementations
of either digital badging systems
or other similar sociotechnical systems.
One of our recommendations focuses on
making sure that the stakeholders
have a sense of ownership of whatever is designed.
Once they feel that it belongs to them
and that they're part of it,
they will continue to use it as opposed to something
that was introduced by outsiders.
If someone comes from outside and says,
"Here, use this technology,"
they're going to look at it
and they're going to think about it
and see if it's worth changing their current operations.
But if they feel a sense of ownership,
they're more likely to engage with it
and continue to use it.
Also, make sure to incorporate the values
that they present to you
and make sure to have those discussions around the values.
We tried to use the pins,
we tried to engage with their culture,
use whatever terminology they use,
tried to become very close.
Which ties into our next one, which is education,
which we focus on teaching them
not only about digital badges and learning from them
about how their program operates,
but also giving them the language
to discuss digital badges with others,
because one of the things that arose often
was nobody else understands what these are.
How do we get the credibility and the understanding?
And so we provided them with the language to discuss that
and now they can educate each other.
They can educate their parents and families and teachers.
And last part is building this rapport.
And over several years, we've built
a very extensive rapport with the students
as well as their supervisors
so that we can have very honest and upfront conversations.
When something is going wrong, they are happy to tell us.
They don't hand-wave it and say, "Oh no, it's fine."
They will tell us when something is going wrong.
They are very honest about it.
So all of these things are very important,
and so those are some of our recommendations summarized.
They're elaborated more extensively in the paper.
So to conclude, I would like to go back
to one of my earlier points,
which is a focus on implementing sociotechnical systems
over the longer term.
Often, we see pilot cities, we see initial implementations
which are very important,
but we don't see the longer term effects.
We don't see what happens after it's been introduced,
what happens over the next several years, how that works.
So we want to focus on how you can use design
and you can continue to use design
through longer implementation
in order to make sure you have a system
that will actually last and actually function
for the community that you're working with.
So thank you so much.
(crowd applauding)
- [Moderator] Questions?
- [Audience Member] Hello, thanks for your talk.
I was wondering what was the purpose of the badges system,
because the only badge you actually explained
was the badge they got a token for
to put on their ribbon or whatever.
And you kind of said you had to force them
to look at the badges system
by putting into the HammerTime breaks.
So why is there no incentive from the badges system itself
to tell the students, "Hey, you got this badge,"
and for the student to see,
"Oh did I reach this badge yet?"
So I didn't quite understand the purpose
and the integration of the badges system
into the other learning system.
Can you maybe elaborate on that please?
- Yes, so in the Pacific Science Center,
they have a number of levels for the students.
So they begin as Discovery Corps Assistants
and they move up through different learning pathways
so the badge system is a way of
digitizing those learning pathways
which were previously analog and on paper.
So this provides a way for them to visually see
which badges they've achieved,
which positions they've achieved,
which they need to achieve
in order to advance in the program.
So that's kind of what this maps onto.
So it's the existing learning pathways in the program.
So this allows them to check them at home,
because previously they had to go to their supervisor
and go get the big binder, the big paper binder,
and look through and see which things they had.
It's called checking out on
once you've been trained in a position
and then you're qualified to do it at the Science Center.
So once they checked out on a position,
previously they had to go check in with their supervisor
and look through a big binder of papers
and see which positions they checked out on.
So now they can go look at the website
and see which ones they have.
And if they have anything that they're missing
or that they need to work on, they can look at it there.
They can send messages to their supervisors
and that sort of thing.
And in terms of them checking it,
it's not that they have to.
They have the option to check it during HammerTime,
so it's not, "You have to do this."
It's that, "Oh, reminding you that this is an option
"for you to do."
It's to raise awareness and allow them to access it
and incorporate it as their schedule.
They're not being obligated to do so,
it's just a reminder like, "This is here,
"maybe check on it occasionally."
You don't have to check it constantly,
it's just, you wanna see which badges you're missing
and you wanna see what you need to work on.
- [Audience Member] Okay, thank you.
- [Guest] Thanks so much for the talk, really interesting.
One thing I was very curious about is that
it's been shown when the gamification's been implemented
in different systems or in online communities
that it can have a motivating factor,
but some of the core components like tables and badges,
if they're not used very carefully,
can actually distance and divide people
in the initial kinda bump in interest,
but if they're not properly managed,
it can drop off massively.
Since you've been doing a long-term study,
have you seen that or noticed things
that would make it last longer?
Have any of them actually had a negative impact
on the students or participants?
- So for the most part, we haven't seen any negative impact.
I do think it's interesting
because when we interview the students
and talk to them about,
"Oh, do you like having badges?
"Do you not like having badges?"
some of them have brought up that
they love being able to see
because they can see each others' badges in the system.
It's not public for everyone,
so not everyone can go look at their badges.
But other students in the system
can look at each others' badges.
So some of them commented that they don't like
the competitive aspect, and some of them
have commented that they do like being able to see it,
but overall I haven't seen a strong difference in motivation
but we would have to look at that more specifically.
It's partially because a lot of these badges
are part of the program requirements,
so they can see which ones they've earned
and which ones they haven't,
but I haven't seen any particular change in motivation.
Particularly, I've seen a lot of interest
in what badges other people have
and how to earn certain badges,
but not a real shift, not a big peak and then a drop off.
- [Guest] Yes, and just out of curiosity, then.
So the introduction of the badges,
what was your core motivation yourselves for doing it
if it wasn't to drive motivations
in the students and participants?
- So we're looking at different ways
that afterschool programs
and other informal learning environments
can be credentialed and allow students
to give credentials to non-traditional learning methods,
so a lot of the times, students who do informal learning
don't get as much credit for doing those types of things
as they would for doing school and college courses.
So we're looking at different ways
that non-traditional education methods
can be turned into credentials that will allow
these students to add them to resumes
or college applications
and use them as an educational tool
and share their learning across settings.
So there's the gamification element,
but there's also a very strong learning element
that we're trying to incorporate.
- [Guest] I see, thank you.
- Yeah.
- [Moderator] Then, thank you all.
I guess I will do a head start for the coffee break,
but before you leave, please let's thank Caroline
and all our other speakers
for a super informative session.
Thank you so much.
(crowd applauding)