- Two of my, whoops!
Let's start the, oh my gosh, sorry.
Let's start, oh my gosh, okay.
The resolution is kind of terrible.
I think I can get it started though.
Here we are.
Alright, thank you all again for coming.
Two of my co-authors are here in the audience with me,
Anicia Peters and Charles Hill, yeah.
The question that we've been interested in trying
to address for a number of years
is whether software supports a variety of smart users,
and the quick answer is no, not really.
It tends to support some kinds of smart users,
and a whole lot of other smart users
aren't so very well supported.
So today's field study tells you
about how four software teams in the field
without us telling them what to do
used a new process that we've devised called GenderMag,
it's a method, to find gender inclusiveness issues
in their own software.
So we'll tell you about their adventures.
So what's GenderMag?
Well, it's a method.
It stands for Gender Inclusiveness Magnifier,
so there's a little magnifying glass right there
to help you remember that.
Its purpose is to evaluate tools inclusiveness
by people who don't have any kind of background
in gender differences or even necessarily in any kind
of UX work, so even software developers are,
we hope, able to use it.
The scope that GenderMag helps you address is when
the software is intended for some form of problem solving.
So imagine a user trying to solve a problem
and coming to the computer to work on it,
on some sort of problem, a classic example
is debugging tools, decision support software,
spreadsheets, anything where the user
is thinking, and thinking, and thinking,
trying to figure something out
and they're using software as part of that process.
That's the scope that we intended GenderMag for.
We've since found out that there's one other kind
of problem solving software that it also works on.
That's when the user is sitting there trying to solve
a problem, not one they brought to the computer,
but one that actually arose
because they're just trying to use the software
and they have to problem solve to do it.
So it turns out GenderMag's pretty good
at finding issues that happen that way too.
It's in beta, okay?
This is a continually evolving method.
In fact, I just uploaded
a new downloadable version this morning.
We update it about every two to three months.
GenderMag, so that's what it's about.
What it has is personas.
It has four personas, and their whole job
is to bring to life five gender inclusiveness facets,
and you'll get to know a little bit of one of the personas
and a little bit of a couple of the facets shortly.
It brings all those things together
with a gender-specialized cognitive walkthough,
which its whole job is
to basically to marry the thing into a process.
So here's just a little bit about one of the personas.
This is Abby.
This is what Abby looks like, at least in this persona,
but Abby has alternative appearances as well.
Abby can also look like this older woman,
this man, or this younger student.
Abby's job is to present a set of facet values.
And so one of the facets is risk.
And so Abby's version of risk
is that she is risk averse with new software features.
This is risk averse in that kind of situation.
She's worried, not because she's afraid
that it's going to bite her or something,
but she's just worried about wasting her time.
So her time is very valuable to her,
and she doesn't want to spend time
and then not get any benefits from it.
So as a result, whenever she has a task
that she can perform some more familiar way
than using some fancy new feature then she will.
So where did we get this?
Well there are these five facets,
and they all have different values,
and these values are all backed by extensive
empirical and theoretical work.
And so what we did is we chose facets that were backed
by at least five independent empirical studies.
So I'm just going to show you a little bit of one study
of one facet just so you can get the general idea.
So this one was about risk aversion,
and so you can see that, well there were kind of,
it was Likert scale stuff,
and so you can see that these people,
all of these people are pretty risk tolerant
when it comes to technology kind of issues,
and these people over here are pretty risk averse,
and these people are in the middle.
So the first thing to notice about that
is that there's quite a bit of overlap,
ways in which the female answers, which I've shown in pink,
and the male answers, which I've shown in blue,
are overlapped so those people
are pretty much like each other.
The second thing to notice is that
the distributions of these are very different.
That's why it matters for us
to think about some facet values
that are disproportionately used more by people
who identify as male than by people
who identify as female and vice versa,
because these distributions tend to be different.
So the persona's job is to look out
for the interests of these people.
So Tim's job, he's one of our personas,
is to look out for all the people
that are like this, for their interests.
And Abby's job is to look out for
the interests of people like this,
and then the Pats, who are actually identical twins,
by this I mean identical in everything,
except their pictures,
their job is to look out for the rest of them.
So let's just look at that black line that I've drawn there.
Imagine that there was software
that had been built in such a way
that it was really only supportive of people
to the left of that line, okay?
Well it turns out there is a lot of software like that,
because it turns out that a lot of the people
who are building that software
or have built it historically have had a lot of facets
in common with this persona.
In any case if there's software like that
what this says is roughly 75% of the females
are not being very well supported,
at least on the risk spectrum,
and also about 1/2 of the males.
So anything that we do to increase the support
of people who have these facet values helps everybody,
but it also helps to level the playing field
on gender inclusiveness.
So the other thing I mentioned is that we have
a gender-specialized cognitive walkthrough.
You may have been familiar standard cognitive walkthroughs.
The black print is pretty much
a standard cognitive walkthrough.
What we added were just a few minor things.
The first is you choose a persona
when you do a GenderMag session,
and so if the team chose Abby then it would say,
"Will Abby have formed this goal?"
This is to really help you connect
with the persona that you are trying to represent.
The second thing is we added a maybe,
so for all the answers to these questions
you can answer not only yes or no,
but it encourages you to answer maybe.
Why?
Because GenderMag is all about inclusiveness.
So we want to include the perspectives
of everyone sitting at the table
not just the person who can shout the route the most
and get everybody to come around and agree.
The third thing is we make people say, "Why?"
So why they made whatever decision they made,
and it points them back to particular sections
of those personas exactly where the facet values are
to really keep them connected,
not only with the personas but to particular places
in the personas that are what
the personas are trying to bring out.
So this is just one example
of how somebody noticed an issue.
Abby was supposed to do something.
They decided no, Abby would not do that,
and their reason why was because Abby,
one of her facet values said she doesn't like
to learn by tinkering.
She likes other ways of learning.
She's more process oriented.
And so what this male said was,
"Abby is disinclined to push and poke
"and so she's not going to discover that."
So that's how they use the facets.
Okay, so let's take a look briefly at our methodology.
So we had four teams scattered among three agencies,
and we had a variety of people on these teams,
and they were doing the evaluation work, not us,
we were just paying attention.
So one had both males and females,
another one had just males, and so on and so forth.
Their job titles, they were software developers
on all of the teams and some of them also
had managers and UX people.
They used mostly Abby,
but one team also used Tim, as personas.
The software domains were very different,
ranging from travel to a mobile app.
The software life-cycle was also very different,
ranging from 10 years old to pre-release,
and all of these were still active.
And the envisioned users of the software
were also very different.
So to the results.
What did they find?
Well the point of this method
is gender inclusiveness issues,
and I was really kind of blown away
when I saw these results.
These people, evaluating their own software by themselves,
with no background in gender differences,
in evaluating they found gender inclusiveness issues
in 25% of the features they evaluated.
That's one out of four.
Some of this software is 10 years old and still out there.
One out of four.
I was really surprised.
So what's a gender inclusiveness issue?
We defined it as one that they found using the facet value.
So for example if they found something and said,
well, anybody who's risk averse
is going to have trouble with that.
That's the way Abby is so she's going to have trouble,
what that's saying is that issue
is going to disproportionately affect
the Abbys of the world, and similarly on the Tim side,
if they found things that related to Tim's facet values.
So that was the definition of an inclusiveness issue.
Here's one example, Abby was supposed to enter a tag.
They decided she wouldn't do it,
and one of the participants said,
"How would Abby know?
"She prefers step-by-step,"
and another one said, "She doesn't like to tinker.
"She's risk averse," she's not going to do that.
So 25%.
They also used the method to find another 30%
on average of just kind of generic usability issues
like the font's too small, that sort of thing.
So in total 55% of the features
they evaluated they found something.
So we have a lot of rich results
and we don't have time to talk about a whole lot of them,
so I'm going to just tell you a few of the interesting bits.
For Agency G, the first team, the one that found only 14%,
they were the first to use GenderMag,
and furthermore they weren't very excited about it.
The boss told them to do it so they came.
The boss couldn't come though,
so there they were,
trotting along because the boss told them to.
Furthermore, they didn't really
have very much empathy for Abby.
As one of the participants said,
"I don't really have very much experience
"with the operators," so okay, fine.
And GenderMag was at its roughest state
because they were the first team to use it.
So we had lots of, lots of bumps there.
The second team had heard about the first team's experience,
but they'd found out the outcome,
which you're about to find out too,
and they were eager to come.
And I think one of the reasons they were so eager
is that they really, really identified Abby
as being a lot like their operators that use the software.
As one of them said,
"Our users are majority female, risk-averse, few tinkerers,"
and they pretty much described Abby to a T
when they were describing their population.
So how'd it go?
Well so GB, they were the ones that weren't
that excited in the first place,
and they left with kind of the same impression,
"Okay fine, that was fun."
And then they told the boss how their session had gone.
At some point they described an issue,
a gender inclusiveness issue they had uncovered,
and the boss said, "I've seen this with our operators."
And that was very convincing to them,
and suddenly they started to really get excited
about the value they could get in advance.
GS started out excited, stayed excited,
they both used the facets pretty well.
In fact risk and tinkering where the most common,
and perhaps the most important thing is
they realized that really it's all about the facets.
It's not really about gender,
it's all about the facet values
and being inclusive across the range of facet values.
And they discovered it.
They said, "It really wasn't about the gender, was it?
"It's about her facet values."
Company E, a female teammate recruited it,
then they couldn't attend.
They arrived unprepared.
Sounds like a recipe for disaster, but actually it wasn't.
Even though they hadn't prepared,
they quickly turned themselves around
and began to embrace Abby.
This was machine learning software,
and they started out saying, "It's not an accountant,"
but they fixed that and then they said
"She's a bit curious but not too curious, and that's good.
"I like her."
The other guy says, "I am channeling Abby,"
and actually the end of his quote was
"And I do not like this software."
He's talking about his own software.
E also discovered without our help
that it was all about the facets.
They said "It's not about gender, is it?
"It's people differences."
and it revealed things they'd never noticed before.
In fact they said "Once you use this
"it makes it easy to detect these things."
It entered into their intuition.
Two weeks later they'd fixed three
of the issues that they found.
W, by now we have a pattern, was a little ambivalent
about trying it, but one teammate championed it
and by the time they had the session 11 people showed up.
It was too many for the room.
They had to kick some of them out.
Ultimately, they did four sessions,
three of which they did with Abby and one with Tim.
They found inclusiveness issues from both perspectives,
but they found more with Abby than with Tim.
One of the things they said is
"We're all a bunch of Tims, so of course
"it was a better fit for the Tims."
Okay, that means stop.
Wait, wait, wait, I have to say this, okay.
After a few weeks, we checked back and it turns out,
Team W didn't own the software
so they didn't get to make changes,
and so they had to rant and rave
and get somebody else to make changes but they did.
They believed, they kept going, they got them fixed.
Five months later it had spread
among some of their little spin off labs.
10 months later they're looking at company wide usage,
and just starting to explore that possibility.
So there's a bunch of triangulation
which I won't tell you about.
In conclusion this is the first field investigation
of software practitioners, ordinary software practitioners,
using a systematic method
to find gender inclusiveness issues in software,
because if you can't find them, you can't fix them.
Our results as you've seen
are one out of four of their features
had gender inclusiveness issues that they found.
I think a nice way to word this is a quote I stole
from Ashcraft and DuBow in which they said,
"Women in technology do not generally need extra help,
"but the software in which they work does need help."
So I hope you download it, I hope you try it,
I hope you teach with it, feel free to talk to me,
because I think that it takes a village
to really change things, and I'm on a mission right now
to change the world and so I hope you help me.
(applause)
Questions?
- [Woman] Thank you for your presentation.
I was wondering if you could comment a little bit more
on the other facets that you include in GenderMag,
and specifically if you could also address
the base of evidence that you use
and whether or not the evidence is
with people of different genders and usability
in online and software spaces
or if it's in other just offline, real life contexts.
Seems like that would make a difference.
Thank you. - Okay, so that's
a big question.
The first one was what are the facets
and the second one was what was the evidence.
So first, jumping to the bottom line,
we have all of our foundation documents online
for these personas and so if you go to our webpage,
you'll be able to read it all for yourself.
The studies were not all done by me.
In fact, I think that's a feature not a bug.
So every facet is backed by at least five studies,
some of them actually have 10 or 15.
The majority of the studies tend to favor US populations,
but certainly not all of them.
We have excluded children from our claims
of it being the target so all of the studies
that we've used to base this on are adults
from all different walks of life,
and all different kinds of studies,
surveys and lab studies and field studies and everything.
The other facets besides risk aversion,
there's way of learning technology,
there's information processing style, and a couple more.
So these are all cognitive problem solving oriented things.
- [Man] Hi, really good study.
I was interested in the people who championed this
as a technique and bringing people in.
Were there certain things that you saw
were more common characteristics
of those champions for this method?
- That's a really great question.
There seem to be about three reasons
why people championed it.
One of them is just passion for the whole idea of diversity,
so we've seen it about that.
Another is a feeling that the method speaks to them,
and it's like they're in the room,
they see it and they feel like,
"Finally, somebody is representing me!"
And it's safe too, because in the past they may
have been sitting in the room,
all the other people kind of liked some feature they didn't,
but they felt "Oh, it's just me.
"I'm not going to talk about it."
But now they don't have to talk about it.
They can just say "Abby does not like that.
There is no way Abby's going to do it."
So it seems like it makes things safe
and this gets them excited.
The third thing is if they're in a position
where they're trying to target a population
that includes a lot of Abbys,
and they realize that they're not reaching them
and they have no idea why, so that business case,
that lights a fire too.
That's what I've seen.
- [Man] Thank you.
- Yeah.
- [Woman] Thank you so much Margaret, thank you.