(no audio)
- Thank you.
It is my honor to present this paper on behalf
of my co-authors from Purdue and Oregon State University.
We have Claudia and Margaret here in the room today.
So I want to begin by saying good morning.
(audience laughing)
And nothing wakes an audience up like some interactive
question/answer sessions.
I will take it easy on you.
You don't have to stand up, but with a show of hands,
you know a lot of us in this room design and deploy systems
out there that people use.
How many of you believe that usability is important?
(hands rustling softly)
Yay.
How many of you actually do something about it?
You can know usability inspection, heuristic evaluation,
usability testing.
(crowd murmuring softly)
Uh-huh, okay, alright, okay.
How many of you believe that gender inclusiveness is
important with these systems?
(hands rustling softly)
Very good.
How many of you do something about it?
(hands rustling softly)
Uh-huh.
Now keep in mind
and I hope you got a chance to look around
that this audience, we're preaching to the choir, right?
Because this audience self-selected to come to
this conference and attend this session.
But as you can see from the show of hands
there has been an awakening that has been happening
over decades, very, very slowly,
but the sun is still not done rising.
So good morning!
(audience laughing)
Well, so if you're interested in identifying gender biases
(audience member coughs)
and gender inclusiveness issues on the systems you develop
one method for doing so is GenderMag which was developed
by Margaret Burnett at Oregon State.
GenderMag is a usability inspection method
in the cognitive walkthrough family
and you can use it to identify cognitive biases
on user interfaces.
GenderMag rests on these five cognitive facets
and the values on these five cognitive facets
have been shown again and again to cluster
around the two genders that we have data about.
And in this presentation we were going to refer to
these clusters as the Abby values
on these five cognitive facets
versus the Tim values on these five cognitive facets.
Abby and Tim are the names of my cats.
No, just kidding.
Abby and Tim are two of the three personas that
GenderMag offers in their toolkit that you could
download from the website if you're interested
to run a GenderMag analysis.
Okay.
So with GenderMag you take these multi-personas,
the personas have fixed values on these cognitive facets,
but then you can customize their goal and interests
to suit your project.
And then you walk through the system
from the point of view of each persona,
and as a result you identify interface issues
that might not support some values on some cognitive facets.
So you know that those are problems.
Those are usability issues or gender inclusiveness issues
and you know you should fix them.
Maybe you even go forth and fix them.
But what happens then? (audience member coughing)
Does usability actually improve?
For whom?
Is it possible that if you fix an issue
that does not support
let's say the risk aversion value of the facet
you're actually hurting the other side?
You're hurting Tim?
So is it possible that if you use GenderMag
perhaps you don't improve inclusiveness?
Alright?
Or maybe you do.
So it's with these kinds of questions in mind
that we set out to perform this study
which is basically an investigation into
GenderMag's effectiveness.
Alright?
We wanted to collect evidence to see how useful and accurate
and what are the effects of doing a GenderMag redesign.
And we did that, doing, using a three-step investigation.
I'm gonna quickly walk you through each step.
So in the first step we conducted a GenderMag analysis
in the context of Microsoft Academic.
Microsoft Academic is a semantic search service for
scholarly publications.
It is a project out of Microsoft Research
and at the time of the study
I was engaged in a complete redesign of its interface.
So two of our co-authors conducted GenderMag analysis
using the toolkit downloaded from the website.
So we used the Abby and the Tim personas
and they customized them to suit their purposes of
academic search.
Developed scenarios with some the tasks that people would
want to accomplish that would be possible in
Microsoft Academic.
And then walked through these scenarios
broken down into sub-goals and actions
which are specific steps a user could take
to accomplish those scenarios.
And while doing so they marked any problems
with the interface.
And they marked which value of which facet might not be
supported, and wrote some comments explaining why,
as you can see highlighted in orange over there.
So as a result of the GenderMag analysis
the two co-authors identified ten issues
of gender inclusiveness on the interface.
In the second step we performed a facet-driven redesign.
And so this is different from a regular redesign,
and I'll explain how with this particular example
from filters.
On the search engine results page Microsoft Academic
has a long list of filters on the left side
and each filter in the original design had 10 options.
Okay?
Now, for the Tim persona, whose learning style is
more tinkering, rather than process oriented and detailed
and reading everything,
we encountered a risk that he would not read
through the entire list of filters
because it was so long, and he would not even notice
that there's a top institutions filter
somewhere down the page.
For the Abby persona,
and particularly her values on the motivations
and risk facets, Abby is task oriented.
Doesn't have time to mess around and tinker and play around
with technology if she doesn't know what the result
is going to be.
She also tends to be more risk averse.
So we had the publication numbers shown on the filters.
The filters didn't look a lot like filters.
Right?
They looked more like a bar chart.
So for Abby we looked at this facet and we redesigned
by removing the numbers to do our best
to make the filter more look like a filter.
And for Tim we redesigned by shortening the list
the number of items on the list.
And so this shows you how the facets
that we paid attention to us lead our thinking
about the redesign.
In the third step we conducted
Wizard of Oz comparative usability testing
using a high fidelity paper prototype.
We had between subject design,
with original versus the post GenderMag redesign
versions of the paper prototype.
And what we actually counted are action failures.
Okay?
So, again, this is something like task performance,
but there can be several action failures,
several steps towards accomplishing a task.
So that's a bit more granular measure.
Our participants were recruited using criterion sampling.
Of course they had to be users of scholarly search engines,
but not familiar with Microsoft Academic.
And we also looked for diversity in terms of
cognitive facets and gender.
So I want to draw your attention to the facet distributions
which they self-accessed using a questionnaire
that had been used previously in research.
And we can see that only three of our participants are
pure Abbys or Tims.
They scored,
on the Abby, one participant scored on all five facets
on the Abby side,
and two participants scored on the Tim side
on all the five facets.
One of them was a woman.
So basically what this tells us is gender is complicated
and cognitively complicated.
And we see that the other 17 participants had a combination
of Abby and Tim-like values on the five cognitive facets.
So this is important to keep in mind
because we did not conduct the analysis by gender.
We actually compared facet by facet.
And so I want to show you our facet distribution
in the original versus post-GenderMag redesign
was just about even for most facets.
Okay.
So let's dive into the results.
So we have two main questions here, and the first one
is going to be about usability.
Does GenderMag improve usability?
So if we look at the number of action failures
we see that in the redesign there are a number of
action failures dropped by more than half
which means that, yes, overall, usability improved.
But for whom?
So the next question addresses inclusiveness.
And I'm going to walk you through how to read this table
that tells the entire story.
So for example, for the first two issues,
if we look at the first row
we see that we identified a issue attributed to
the Abby value of the motivations facet,
and we redesigned for it.
And in the usability effects we helped usability for Abby
but also for Tim as a nice side-effect.
Another example, same issues, for the learning facet.
We identified issues and redesigned for both Abby and Tim
and we ended up helping both Abby and Tim.
So the question is did we hurt anyone?
Well, there are only two instances where that happened.
In an instance where we redesigned for Abby
we helped Abby, but we ended up hurting Tim.
And another instance where we redesigned for Abby
and our redesign ended up hurting Abby, but helping Tim.
Okay?
However, if you look overall, you see that the redesign
did not hurt the other side of the facet.
It either kept things the same or more often than not
it actually improved usability for both sides of the facet.
Okay.
So let's take an in-depth look to see how we got here
and how these results came about.
Now in the paper we tell the story of all the issues
that we identified.
We talk about the facets that were involved
and the rationale.
We describe the redesign and we show
before and after pictures,
as well as detailed results from the usability testing.
So here we're just going to look at the results
of the usability test for one of the issues.
And this one we're going to read it by column
to show you how we did the facet by facet comparison.
So for issue four we see that in the original condition
six participants experienced action failures.
If we look at the motivations facet,
we see that of those six participants,
four had Abby-like values on this facet,
two had Tim-like values.
Now let's look at the same facet in the redesign,
and what we're learning here is that we helped Abby.
There were no action failures with the Abby-like,
but Tim remained the same.
If we look at another facet, information processing,
same story.
Same for the original.
However, in the redesign
we actually helped both Abby and Tim.
Okay?
So now, going off to the next question:
Is GenderMag accurate?
So when we think about accuracy one of the questions we ask:
Does it actually identify issues accurately?
So if the analysts say,
"this is going to be an issue that needs to be fixed,"
do the participants in the usability test
actually experience that issue?
So in our case
100% of the issues
that the analysts identified
actually happened to the participants.
Alright?
Now this is consistent with previous data
for GenderMag and cognitive walkthroughs
that showed that they're pretty good in terms of
true positives.
Rates are usually 96% and above.
But there's also the question of missing issues.
Were there any issues that the participants encountered
that the analysts didn't point out?
Now if we look at the previous studies we see that
that happens 30-70% of the cases with cognitive walkthroughs
In our case there was at least one instance,
one blatant instance, where this happened.
And we tell the story of that in the paper.
Another thing to note is that the analysts,
they were graduate students in HCI,
but they were not GenderMag experts.
This was their first time conducting a GenderMag analysis.
And in spite of that their accuracy rate was very high.
Another question that we ask in terms of accuracy is:
Does it identify facet biases accurately?
So if an analyst says, "we believe this is going
"to be an issue for the Abby side of the motivations facet,"
Do people who score on the Abby side of the motivations
facet actually encounter that issue?
In our study this was indeed the case in 75%.
(audience member coughing)
That's 15 of the 20 facets that were identified
by the analysts.
So, once again, that a pretty good rate.
Okay.
So we've seen the results and the yeses over here.
One question that we still haven't addressed is:
Well, how about gender?
Because all these analyses were done by facet.
What happens with gender?
So let's look at our participants again.
Just like in the other studies
we see that women skewed toward Abby values on the facets
and Tim,
and men towards Tim values.
So then if we do the analysis by gender we see that
in the original condition
women experienced twice as many action failures as men.
However, in the redesign,
not only did action failures reduce,
but the level,
the playing field was leveled.
This is where I was supposed to do the mic drop,
but I'm not allowed.
(audience laughing)
So I want to briefly touch on implications for design.
Is GenderMag effective?
(audience member coughs)
Uh-huh.
Okay.
Another lesson that we're learning
is about the facet-driven redesign.
Using the GenderMag facets to inspire our redesigns.
The next story is that it is, but it is not about gender
and this is a story that we've heard from
pluralism in feminist HCI.
And finally I want to leave you with the thought that
cognitive diversity is gender diversity and so if you
support cognitive diversity you're going to support
gender diversity as well.
(applause)
- [Woman] Thank you.
Do we have any questions?
Jessica.
- [Jessica] Hello.
So I, you said at the beginning that
there were three personas.
So the Abby and Tim.
What's the third persona like in that case?
- So the third persona is a combination of the five facets.
Has some Abby values and some Tim values.
But we used, for clarity and simplicity,
the ends of the spectrum, the Abby and Tim.
And the middle person's actually their Pat
Patrick or Patricia.
But we opted for the, to keep things clear,
for the other two.
- [woman] Are there more questions?
- I think there's one.
Oh, or not.
- [woman] Okay, while we're waiting to get the mic
I have a quick question, which I'm dying to ask,
because I, I think this is, you know,
I've read a bunch of GenderMag papers,
this is really important work,
but as we know from intersectionalist and queer theory,
we aren't just Abbys and Tims, we have a whole range
of nuanced, often trans and gender queer identities
and I know that you guys have engaged with that elsewhere
but you haven't had a chance to in your talk
so could you explain how you could use GenderMag
and be inclusive to LGBTQIA gender identities?
- Sure.
Thank you so much. Thank you so much for this question.
And one thing that we struggled with, you know,
a lot of these cognitive facet measures,
they're only reported for two genders,
but if you look at how we analyzed the data
and how we looked at each individual, right?
One individual can have a combination of values, right?
So they could be three Tim, one Abby or four Abby, one Tim.
And this is not necessarily related
to the gender that they report, right?
'Cause we saw we have one person with all five Tim values
who identified as a woman.
Okay?
And so in this sense, if we, and this is why
we wanted to do the analysis by facet, rather by gender.
We only had two genders represented, right?
And we didn't ask about sexual orientation in our study,
but we believe that if you look a the individual,
and the individual combination of facet values,
this is how you can begin to address that.
But of course it would be really, really interesting to
extend the research on these values and see how, you know,
the variety of people and genders might score.
Or even just keep in mind that each individual
can be a combination.
- [woman] That was a fantastic answer
to a really mean question.
Next question.
(no audio)
- [Audience Member] I have a similar question, in the sense
that thinking about values in gender there's a lot of
intersectional issues in terms of race, ethnicity,
nationality, things like that.
Is that considered at all in these sort of evaluations?
- So I don't know that I can or should answer in general
for these sorts of evaluations, because I'm not really sure
what sorts of evaluations you're referring to
or weather you're referring at the previous literature
that informs GenderMag.
I can only tell you that in our present study
I described the participants and these are
the participants that we worked with.
But I'd be happy to talk to you more
about previous research and literature.
And of course, Margaret over here,
would be happy to answer that question for you afterwards.
- [Woman] Thank you very much.
We're gonna have to go to our next speaker.