[INTRO MUSIC]
ABBY SCHMIEDT: So my name's Abby.
I work with the Kids Coding team here at Google,
and I'm going to talk a little bit about keyboard navigation.
So first off, I'm going to give you
a little background about why we chose to start
with keyboard navigation.
I'm going to give you a quick demo of what we currently have,
and then the rest of talk, I'm going
to talk about different ways that people
can change and configure keyboard navigation.
So one of those ways is being able to change
the key mappings, so what key maps what to what action.
So opening the tool box currently
maps to T, just because I chose that.
There is really no good reason.
And then workspace navigation is how
a user navigates the different inputs, fields, connections,
and blocks on a workspace.
And then I'm going to show different ways
that people can change this for maybe different audiences.
And then finally, I'm going to talk about some future work
for our keyboard navigation.
So why keyboard navigation?
So keyboard navigation is kind of our first step
in creating an accessible Blockly experience.
Making Blockly completely accessible
is a very large and very daunting task,
so we wanted to pick something that was smaller
and something we could more easily bite off.
Second off, it's useful for sighted users or power users
who want to use a keyboard.
It's definitely useful.
And it's potentially useful for people with motor impairments.
So our goal for a keyboard navigation
was create an easy way for developers and researchers
to test keyword navigation.
It wasn't to create a perfect keyboard navigation setup.
We're really aware of the fact that this is a large problem,
and we don't really have the perfect answer for it.
So we really wanted to empower other users
and other researchers to test this problem along with us.
So with all of that said, I'm going
to do a quick demo of what we currently have.
So in order to get into the Keyword Navigation mode,
I'm going to Shift-click on the screen.
You can also get there by hitting Control-K.
So this will put a cursor on the screen.
It's that blinking red line.
Next, I'm going to hit T, and that's
going to open the toolbox and the flyout.
So now I'm on the toolbox.
I'm going to use the WASD keys to move around the flyout
and toolbox and also be using the WASD
keys to move around the different connections and stuff
on the workspace.
So I'm going to hit D. That's going to go to the If block.
Hit Enter.
Now I'm going to navigate to the next connection.
I'll go into more information about how I'm navigating
to the different connections and inputs later on, but right now,
good to know that I'm going to the next connection point
that I want to enter our block to.
So I'm going to hit Enter, and that's
going to mark the location.
So you can see from the blue line that showed up
that location is marked.
That's my insertion point.
So I'm going to hit--
making so much noise.
Cool, so I'm going to then enter another block on the game.
So I'm going to hit this one and enter one more.
Cool.
And then just for demo purposes, I'm
going to add another block to the workspace.
So I've added a few blocks on the screen.
Now I'm going to navigate to the block I just added
and import that into my stack.
So in order to do this, I'm going
to navigate to the stack level.
So I can now navigate to the next stack,
and I'm going to find--
actually, I'm going to back up a second.
I'm going to mark my location that I
wanted to insert the block to, so that's the blue line.
And then I'm going to navigate to the block
that I want to insert.
So once I navigate here, I'm going to put I,
and that's going to insert this block into the stack.
So the last thing that we can do with keyboard navigation
is we can disconnect blocks.
So I'm going to hit X key, and it's
going to disconnect the blocks.
So that was a very quick demo of what we currently have.
The rest is just going to be me breaking down the demo
and talking a little bit more about it.
So first thing we're going to talk about
is key mappings and actions.
So we're going to talk about what
key mappings and actions are and then how we can modify them.
So a Blockly action is meant to represent a user's intent.
So if a user wants to go up on the screen,
whether they're in the toolbox or on the workspace,
that will map to the same key.
And then actions hold their name and description
of what they do.
So currently, the actions we have are a Previous,
Out, Next, and In.
These are mapped to the WASD keys.
Insert, which is when I connected
the second block on the workspace into the stack--
that's mapped to the I key.
Enter, which creates that blue line we saw on the demo,
is for Mark.
X is to disconnect.
T is for the toolbox.
Escape is closing the toolbox, and Control-K
is toggling the keyboard navigation on and off.
I'll pause there.
Any questions or anything?
AUDIENCE: This is probably wrong,
but is this on [INAUDIBLE]?
ABBY SCHMIEDT: Yeah, so it's going to be in our Q3 release.
So that's coming up on Monday.
But it's hosted on Rachel's site right
now, so it'll be available to play with, yeah.
So now to modify key mappings.
So like I said, we wanted to make it really easy for people
to change the key mappings.
So there's two methods.
One is setActionForKey, and the other is createSerializeKey.
So the very thing we do is create the serialized key.
It takes in the key code and a list of modifiers.
We currently have an enum with modifiers.
I think it's Control, Command, Shift, and Alt, I think,
are the four keys that we currently have.
And I was just told to look up instead of to the side.
So once we get that key code, then we
can set the action for the key code.
So there's a list of actions in the navigation file,
and you can pass it the key code you
created from createSerializeKey and the action you want.
So the reason it's important to be
able to modify the key mappings is that, for accessibility,
a lot of people have their own setup already.
So it's really important that we don't interfere with that
and we allow them to make this something that
will work for what they already have set up.
So now we're on to workspace navigation.
So workspace navigation is how a user navigates
the different fields, inputs, blocks, connections,
and workspace coordinates.
So these are all the different elements
that a user might want to navigate.
So I'm just going to show a few of those elements.
So right now this is showing that previous connection.
If I go in, we're showing the input.
If I go out and down, this is the next connection.
We also have the stack, and then this is showing the workspace.
So there's lots of different elements on the workspace
that a user needs to be able to navigate in order
to use Blockly properly.
Yeah?
AUDIENCE: [INAUDIBLE]
ABBY SCHMIEDT: Like moving around?
AUDIENCE: Yeah.
ABBY SCHMIEDT: Yeah, currently it does not.
It's going to.
We just need to figure out the best way to do it.
So I think one of the ways recommended
was using Shift-WASD to move around the keys,
so it just needs to be implemented.
Great.
So in order to represent all of those different fields, inputs,
connections, blocks, and stacks, we created an AST.
So the AST holds all of these values
and is a structured way for us to look at them.
So this is an example of the AST.
It's really not the prettiest picture, but I added colors,
so hopefully that will help.
There's currently four levels.
There's the workspace level.
Underneath that is the stack level, and this
holds all of the stacks on a workspace.
Under that there's the block level,
so that holds all of the blocks and then all
of the previous and next connections on that block.
And it holds all of the ones that
are nested at the same level.
So as you can see, A and B are both--
they're not nested, so they're at the same level.
And then finally, the fourth one is inputs and fields.
So anyone have questions on this gorgeous photo?
No?
OK.
Yeah?
AUDIENCE: [INAUDIBLE] like a visual program [INAUDIBLE]??
ABBY SCHMIEDT: Yes, that is a great question and leading
into what I'm going to say next, so perfect.
Thank you.
So real quick, the AST is created using AST nodes.
They hold the location, so the field, connection, workspace,
coordinate, and they also hold the type.
And then and AST node can go to the next In, Out,
or Previous location from where it currently is.
Cool.
So the cursor sits on top of the AST
and controls how the user navigates around the AST.
So the AST is mainly a structured way for developers
to look at elements on the workspace,
but the cursor is what they use there is actually going
to be able to navigate with.
And we need this because the user doesn't necessarily
want to go to every single spot on the workspace
or on the different blocks, so the different cursors
allow the developer to control where the user is going to go.
So that's why we need the two different layers, yeah.
So the cursor currently holds the current node that it's at.
It has a way to set the current node,
and it has Next, In, Previous, or Out.
These are something that a developer can override
to create multiple different creatures that
do different things.
So I'm going to show maybe an example of that.
There is my gorgeous keyboard nav demo
page with lots of text.
So we have a default cursor, and this default cursor
navigates really similar to how the AST is created.
So I Shift-click.
I can go to the next item, which goes down in the tree.
If I go in, this will go to the next input.
If I go in again, it'll go to an output.
If I keep going in--
at some point I keep trying to go in, and I can't do anything.
So I'm stuck here.
So this might be really good for someone
who knows more about Blockly, but for someone
who's just starting, this is really frustrating.
Just hitting D and not having it do anything is really annoying.
So we also created a basic cursor.
So this does a pre-order traversal of the tree
and allows us to go to the next or previous location in it.
So I can get to everything I want in this stack of blocks
by either using the D or the A key.
So if I hit D, I will just--
well, probably easier to show it down here--
get here, and I will keep going.
So the cursor is nice because it allows
you to create different cursors for maybe different audiences.
And then finally, future work--
wow, I think I went really quickly.
But future work-- we want to provide an easy way
to change the cursor rendering.
So the red line and blue mark that you guys on the screen
is not necessarily easily changed.
It can be changed right now, but we
want to provide APIs to make that easier.
We want to support multiple workspaces,
and then we need to be able to navigate
all the non-block components on the workspace--
so the trash can, the workspace comments,
the icons on the block.
And finally, we'd really like all of your feedback.
So we currently haven't had a great chance
to test this with users, so we'd really
like you to tell us what is great, what is not great.
And it can be found on our release candidate
playground, which Rachel wrote on the board back there.
So you're more than welcome to play around,
and it has the demos for what we did, as well.
And Sam's modifying it.
But yeah, so feel free to take a look at that later.
Any questions?
Yes?
AUDIENCE: So after playing with it for a while yourself, what
[INAUDIBLE]?
ABBY SCHMIEDT: I'm not sure exactly what factor.
It doesn't make me faster when I'm just
trying to add blocks and stuff to the workspace.
I haven't tested it with, like, really large workspaces
and lots of blocks, so that would be,
actually, really interesting to try.
AUDIENCE: But you can [INAUDIBLE]??
ABBY SCHMIEDT: Yeah, for me, at least, for the simple things
I'm doing, yeah.
Yes?
AUDIENCE: Yeah.
[INAUDIBLE]?
ABBY SCHMIEDT: Stuck with those?
AUDIENCE: Yeah.
ABBY SCHMIEDT: So the question was
about actions and whether they can be expanded.
We don't have a great API for adding new actions,
but in the accessibility hackathon tomorrow,
it does show you how you can add new actions
and how you can make it do exactly what you want.
Yeah, great.
Yes?
AUDIENCE: [INAUDIBLE]
ABBY SCHMIEDT: That is actually my project for next quarter.
Yes, I haven't thought of it a lot.
I think it's a really good opportunity.
It'll be a lot faster to move around and search
for different things, so yeah.
Sam, yes?
AUDIENCE: [INAUDIBLE]?
ABBY SCHMIEDT: Yes, so the question
was why we chose WASD for navigating around
instead of the arrow keys, and it
was because arrow keys are generally what
people use for screen readers.
So we wanted something that didn't collide
with that, which is-- but yeah, it can get a little confusing
sometimes.
I definitely tried to use the arrow keys
and been like, why did I break everything?
Why does nothing work?
Yes?
AUDIENCE: So I don't have any questions.
[INAUDIBLE]
But I'm kind of wondering [INAUDIBLE]??
ABBY SCHMIEDT: Yes, that's a good question.
So you're asking who the target is for keyboard navigation?
Yeah, so I think eventually the target
is going to be people who have limited sight,
and we want it to be able to work
with some kind of screenreader or some kind of ARIA
live region and read out what's on the blocks.
That's a long way out, though, currently.
So another use case is for power users
who want to just be faster, and the third
is case which we haven't explored that much
but has potential is for people who have motor impairment.
So they can't use the mouse.
There is ways that you can have the cursor just move around
the different blocks, and they could maybe use a pedal
or something else, too.
[OUTRO MUSIC]