[MUSIC PLAYING]
ABOUBAKAR MOUNTAPMBEME: My name is Aboubakar Mountapmbeme.
I'm from Cameroon.
I'm a second year PhD student in the Department of Computer
Science at the University of North Texas,
and my advisor is Dr. Stephanie Ludi.
And I thank her for giving me the opportunity
to be here today to represent our team
and talk about our work.
My research interest lies in the area
of HCI software engineering and machine learning.
And concerning Blockly, I've been working with Blockly
for the past one year, and I have enjoyed the experience
so far.
So before we dive into the presentation,
we will briefly go through the outline.
So we will talk about the motivation, why we
are making Blockly accessible.
We will discuss the approach that we
follow to add accessibility to Blockly.
And we will have a quick demo of what
we have accomplished so far.
From there we will discuss the challenges
and some possible feedback solutions,
and we will have a brief look at our current and future work.
So why are we making Blockly accessible?
First of all, we know that visual programming languages
are very successful in introducing novices
to programming, mainly because, unlike text-based languages,
they don't have this heavy syntax that imposes
some constraint for new users.
So that is why we feel if Blockly is accessible,
then it will be equally useful for users
of assistive technology.
Then we also have this desire to make programming inclusive
and broaden participation in computational thinking
activities that involve block-based programming.
So, before, there was one separate version of Blockly
that was designed for users of assistive technology.
And having two separate systems might not really
encourage collaboration for students
who have sight and students with visual disabilities.
So we feel having one system which is accessible
for everyone is a good way of promoting collaboration.
So these are our motivations, and now we
will go through our approach.
So, basically, to make Blockly accessible,
it simply involves adding support
for assistive technologies, because we know people
with visual disabilities, they rely on alternative devices
or alternative interaction techniques
to communicate with the computer.
So if we add support for these devices
to our Blockly environment, then everyone
will be able to use and benefit from Blockly.
So we also have to consider the ease of manipulation
of some of the blocks, ease of interaction.
So we also look at how we can redesign some of the blocks
so that it can be easy for those who
rely on assistive technologies to interact with the system.
So as we go through the presentation,
we will go deeply in each of these steps.
So the Screen Reader-- so we know
that the screen reader that will read the content of the screen
to the user.
And Blockly being visual in nature
relies on visual metaphors to make programming easy and teach
concepts to the users.
But along these visual blocks, we have natural language
descriptions of the blocks, and if we
can extract this text that describes the blocks
and then send it to the screen reader buffer,
then it will be able to read that as output to the user.
Luckily, we have Blockly, which is developed
in CSS, JavaScript, and SVG, so we
can follow the accessibility guidelines by the W3C Web
Accessibility Initiative Group, known as the Accessible Rich
Internet Applications.
So if we follow the ARIA guidelines,
then we will easily have Blockly be usable
by assistive technology users because accessibility
in web application is not automatic, right?
So you have to follow--
you have to design your application with accessibility
from the beginning.
So if we follow these guidelines and include them
in our Blockly, then assistive technology users
will have access and easily use the application.
Now, how do we enable keyboard support?
Now, we know that Blockly is mouse driven,
so all the actions on Blockly, the interactions,
are through the mouse, but people
with visual disabilities, especially those who are blind,
they use the keyboard as an alternative method
of interacting with the computer.
So for us to have Blockly, for Blockly to be fully accessible,
we need to support--
to add support for the keyboard.
And luckily, the Blockly team have done a tremendous work,
and they already have the support
for the keyboard interaction.
Looking at this point for keyboard interaction,
we have to identify the operations that are involved
in interacting with Blockly.
And we have two main operations, which is
Drag and Drop and Navigation.
Drag and Drop, like dragging the block from the toolbox
to the workspace, and then Navigation, which
involves going through different blocks,
it may be any particular category on the toolbox.
So the problem comes here, because when
you are doing this interaction with the keyboard,
you can have different approaches.
For example, Drag and Drop can be performed in two ways.
So if you want to drag a block from the workspace
to the toolbox using the keyboard,
for example, you can first select a block in the toolbox,
then you select the location on the workspace,
and you drop the block.
Alternatively, you can select the location in the workspace,
then you go to the toolbox and select the block,
then you drop the block.
These are some of the things that we
have to take into consideration when designing accessibility
for the keyboard.
And you cannot tell which one is better or which one is good.
It's only through empirical studies with participants
and users that you can say, OK, maybe
users prefer this method than this one.
So these are the things that we have to consider.
And also we have to be careful when selecting the keys that we
use for interaction in order to avoid conflict
with existing applications.
So in our system, we mostly use the WASD keys
as the main keys for navigation.
So this is because the WASD keys, unlike the arrow keys--
so we choose to use the WASD keys, unlike the arrow keys,
because the arrow keys are already being
used by the screen reader.
So if we use this one, it will conflict with the screen reader
applications.
And WASD keys are also easy to access on the keyboard.
So that's why we chose these keys.
So these are the things you have to consider
when adding a keyboard interaction to Blockly.
Now, another aspect is redesigning some of the blocks
to ease manipulation, because some of the actions that
can easily be performed with the mouse,
when you go to the alternative interaction
methods like the keyboard, it may require extra efforts
for the users.
So we have to redesign these blocks
so that those who use the keyboard
can easily build their programs.
So we have these mutator blocks that we identified.
And mutator blocks, as we can see here
on the top left of the--
on the top left corner of the screen,
we have this diagram that shows an if block.
And the if block, we see it has a mutator on it.
This mutator, when you click on the mutator,
you have an additional dialog that
will allow you to construct additional else if and else
blocks.
So we feel that this operation for somebody
who is using Blockly through the keyboard
will require extra effort and might
impose some memory constraints.
So that's why we redesigned these mutator blocks,
so that for each mutator, we have
a separate block in the tool box that the user can use.
And this also helps maintain consistency,
because the user might have in mind
that, OK, for me to construct my program,
each time I need a block, I have to go to the toolbox,
then get a block, and then add to the workspace.
But if we have mutators for some of the blocks
and not for others, so the user has to keep track in their mind
that, OK, which block has mutator, which block
does not have mutator, and this can impose some memory
constraint and make learning a little bit
harder than we might expect.
So similarly, we also have the mutator blocks
for the functions.
For you to create a function with Blockly,
you have to drag the template block to the workspace,
and then if you want to add extra parameters,
then you have to go through this pop-up dialogue.
But we also redesigned that so that our template just takes
a field, which is a number.
And that number will--
oh, sorry.
That number indicates the number of parameters
that you want for your function.
And once you add the number of parameters,
then you'll get the function generated for you.
So these are some of the considerations
that we have to look into when we want
to make our system of Blockly or any other block-based
environment accessible.
Now we will go through a quick demo
of what we have done so far.
And while I'll be doing the demo,
I will not talk because the screen reader will be on.
So if I talk to you, you might be able to hear the screen
reader, not hear me, right?
SCREEN READER: Blockly interface dash Google Chrome
ABOUBAKAR MOUNTAPMBEME: So let's set our demo.
SCREEN READER: Address and search bar--
[CLICKS]
Selection--
[CLICKS]
D colon slash Blockly project slash Blockly slash Blockly
slash accessible Blockly--
[CLICKING]
Blockly--
[CLICKING]
Address and search bar.
Blockly level 1--
[BEEP]
ABOUBAKAR MOUNTAPMBEME: OK.
SCREEN READER: Loops 2 of 8, level 1.
Logic 1 of 8, level 1.
[CLICKS] If-- left paren--
[CLICKS]
If a.
[CLICKS]
Edit mode entered now.
[CLICKS]
Top connection.
[CLICKS]
If connection.
[CLICKS]
Do connection.
[CLICKS]
Bottom connection.
[CLICKS]
Do connection.
[CLICKS]
If connection.
[CLICKS]
Tree view.
[CLICKS]
Loop--
[CLICKS]
Math 3 of 8, level 1.
[CLICKS]
Disabled--
[CLICKS]
Disa-- disabled square root of left--
[CLICKS]
Disabled--
[CLICKS]
Left paren.
Number right paren is even block.
[CLICKS]
A is even.
A is even on the--
[CLICKS]
Edit mode entered now.
[CLICKS]
Tree view.
[CLICKS]
Loops 2 of 8, level 1.
[CLICKS]
Math 3 of 8, level 1.
[CLICKS]
Number block
[CLICKS]
0 is even.
0 is even alerts.
[CLICKS]
0.
[CLICKS]
Edit mode entered now.
[CLICKS]
At 1--
[CLICKS]
Blockly interface document.
[CLICKS]
10 is even.
[CLICKS]
If 10 is even.
[CLICKS]
Edit mode entered now.
[CLICKS]
Top connection.
[CLICKS]
If connection.
[CLICKS]
Do connection.
[CLICKS]
Tree view.
[CLICKS]
Loop--
[CLICKS]
Disabled text.
Disabled--
[CLICKS]
Print left paren.
[CLICKS]
Print a.
[CLICKS]
Edit mode entered now.
[CLICKS]
Top--
[CLICKS]
Text connection.
[CLICKS]
Tree view.
[CLICKS]
Loops 2 of 8, level 1.
[CLICKS]
Math 3 of 8, level 1.
[CLICKS]
Text for a--
[CLICKS]
Text block.
[CLICKS]
Left quote, right quote.
[CLICKS]
Edit mode entered now.
[CLICKS]
Edit mode entered now.
[CLICKS]
Edit
[CLICKS]
D.
[CLICKS]
W.
[CLICKS]
Blockly interface doc--
[CLICKS]
Print left quote, hello right quote.
[CLICKS]
If 10 is even.
[CLICKS]
Edit mode entered now.
Edit mo--
[CLICKS]
Tree view.
[CLICKS]
Loops--
[CLICKS]
Repeat left--
[CLICKS]
Repeat a times.
[CLICKS]
Comment colon graphic.
[CLICKS]
Comment colon.
[CLICKS]
Edit mode entered now.
[CLICKS]
Tree view.
[CLICKS]
Loops--
[CLICKS] Repeat--
[CLICKS] Repeat while or--
[CLICKS]
Repeat while a.
[CLICKS]
Repeat a times.
[CLICKS]
If 10 is even.
[CLICKS]
Print left quote hel--
[CLICKS]
If 10 is even.
[CLICKS]
If 10 is even.
ABOUBAKAR MOUNTAPMBEME: So--
SCREEN READER: If 10 is even.
ABOUBAKAR MOUNTAPMBEME: This is the demo
of what we have accomplished so far
and how you can construct your program using the keyboard
and the screen reader.
So we will switch back to our presentation.
SCREEN READER: Blockly so--
note-- note--
ABOUBAKAR MOUNTAPMBEME: Let me turn off the screen reader.
SCREEN READER: NVDA.
NVD-- Exit.
Exit NVD-- OK, OK.
[EXIT TONE]
ABOUBAKAR MOUNTAPMBEME: OK.
So now after the demo, are there any questions or--
OK.
So some of the challenges and feedback solutions
that we have.
One of the first challenges that we face will be--
we'll face is to avoid conflicts with existing applications
and the screen reader, in particular.
So if you are not careful in choosing your keys,
so when you--
if you choose your keys, and it shows
that it will conflict with existing applications,
then you might face a lot of difficulties.
Then another challenge is choosing a navigation style
for the keyboard.
So, as I earlier mentioned, you cannot tell which one is good,
which one is better.
So it's only through empirical studies,
and that leads also to our next challenge that we face,
getting enough participants to conduct the empirical studies.
We also had problems keeping up with the updates from Blockly.
Each time they had to update the main repository,
we had to pull the code, and that created some problems.
And we had to go through some extra work
to fix our code so that it can work.
Then some of the feedback solutions that we have,
we will--
it's related to the keyboard API that they have developed,
so given that we can have multiple ways of interacting
with the system through the keyboard for one
particular action, if they can make the keyboard API such
that you can easily implement your own custom navigation,
so that way you can test different navigation styles
and patterns, and then you will be able to maybe tell
more which one is preferred by the users
during the empirical studies.
Then we also plan on adding audio cues in order
to provide more feedback information to the user.
Like, if you can have audio cues that
indicate the start and the end of blocks
or audio cues that convey structural information,
such as nesting.
Now, we will discuss about our current and future
work, which I already referred to some of them
in our previous slide.
Currently, we have two surveys that we have designed.
One will be to better understand the mental models of users
when navigating the code structure with Blockly.
And we also plan to explore the differences
that novice programmers and experienced programmers
might have with this system.
And we have as future work to investigate
the use of audio cues to provide more contextual information
for the users.
So for more information on our work,
you can consult these publications
as well as go to our GitHub repository,
where you will get what we have implemented.
Thank you.
Are there any questions?
[MUSIC PLAYING]