(applause)
All right, thank you Professor Bernstein,
for the introductions.
Hey everyone, my name is Alok Mysore, and I just graduated
with a Masters from UC San Diego,
where I worked with Professor Philip Guo on ACI Research.
So I am applying for a PHD about a year down the line
so woohoo!
So for those of you who are here looking for
an exciting presentation on like either on Cloud or Seri
I'm really sorry to disappoint that my presentation
is on neither of those.
So about a year ago I was on the same stage
in Quebec, Canada at UIST 2017, presenting my paper
on generating...
on generating technical tutorials
via operating systems via activity tracing.
This was a system we called Torta, and Torta enabled
creating quality tutorials super easily.
But there's already millions of tutorials
that are already out there,
as Professor Bernstein mentioned a few minutes ago.
And we wanted to see how we can improve those tutorials.
So we first looked at the tutorials that already exist.
So for one thing, a lot of them were obsolete,
either because they had differing computing environments,
such as differing operating systems.
Or they were obsolete because of
version deprecations and things like that.
The second thing to note was,
there was usually no usage data,
and when learners had problems
that they found in the tutorial,
there was no way that they can communicate
that to the page of the tutorial,
apart from writing that in a comment section
that maybe no one might ever see.
The third, and most important point to share
is that a lot of times, seemingly proficient authors
created tutorials that were too high level
for the learners to understand.
So this is super common in the computer science world,
where very common command name tools like
cat or grip, which are understood by experts,
are points of difficulty for people who are new to it.
And this is what we call the expert blind spot effect.
So with this project and with this system,
we wanted to see if we can give instructors
effective feedback about learner behavior.
So, to give you a bit of context
about my own investment in this project,
I was TAing this Introduction to ACI course,
which was also the one that Professor Bernstein created.
And we have a lot of material that already looked like this
and we had almost absolutely no idea
on how users were using them.
We didn't know if people were actually following these
or if they were going to start call flow
or if they were using these to sort of index through cycle.
So we had no idea.
So we want to be able to record these learners
looking at out tutorial and then sort of visualize it
on top of the existing tutorial.
And this is why we built our system which we call Porta,
which is short for Profiling Operating system
Recording for Tutorial Assessment.
So Porta is a system to record learners
while they look at the tutorial and perform actions
on things like IDEs, terminals,
text inserters, browsers etc.
And then visualize all this data on top of
the existing tutorial, so that creators such as myself
can gather insight from this.
So firstly, what do we want to record?
So this is typically the user case that we saw.
So a learner usually looks at a browser
where there's a tutorial
and then performs actions on either an IDE
or a terminal or a text inserter etc.
So we want to record them while they follow the tutorial
on the tutorial webpage, which we call the browser events.
And we also want to record the actions they took
while following the tutorial,
which we call the system events.
So when we look at browser events,
we want to look at them opening all webpages.
So, for example, if the tutorial did not have enough context
on the grip tool on the command lane,
they might go to Google and search for it,
or if they want see how the iterators on Python work,
they might go to Stack or Flow for that.
We also want to, we also recorded them playing or pausing
embedded videos on the webpage.
And we also want to figure out the learner's focus area
without having to invest in expensive eye tracking gear.
So how do we figure out what learners are focusing on?
So in this example tutorial that you see here,
the learner has their cursor on a block of code,
a block of lower code in Java.
And there's a pretty good probability that the learner
is looking at that block of code right now.
But there's also a good chance
that they were looking outside where the mouse is pointing,
and in all likelihood, they're looking
at the center of the page rather than the edges.
So if we add both of these components we get
an approximation of the user focus.
And in our experience, this is actually pretty accurate.
It was not 100% but it was actually really accurate
in terms of what the user was looking at.
When we looked at system events,
we recorded all of the learner's screens
while they performed the actions and followed the tutorial.
We also recorded them copy-pasting things
and other clipboard actions.
For web specific tutorials, we recorded them opening
the Chrome developer tools and also JavaScript errors
that occur on the browsers.
We also, the last parts that we recorded were them running
certain terminal commands and also toolchain actions,
such as gcc, make, javac etc.
And this worked both on their own local machines,
or if they researched into our remote server
when they were following the tutorial.
So you might ask, why do we record toolchain actions
separately from just terminal commands?
Isn't running GCC just through the terminal,
can't we just do something like that?
So we wanted to record these things separately
cause often times if you are using an IDE,
such as a Eclipse or PyCharm or something.
These binary such as javac, make etc. remote by the idea,
not the terminal, so we want to record those things.
We also want to log the source code that was compiled
by these toolchain binaries.
And we also wanted to record things
like stdin, stdout, stderr files.
So what we built essentially was a shim
that replaced the original binary
in the dollar path variable.
This shim was essentially,
this shim was identical
to the original binary and internally denotes
the original binary itself.
The only difference is that it logs
the command line arguments, the standard input,
output, and error files onto a log file.
You also see that, we also attach
a DTrace model to the original binary.
So lets go through an example of why we need this.
So suppose the learner is going through
this particular java program with two files,
Client.java and Lib.java.
Suppose Client.java imports certain functions from Lib.java.
When compiling, you usually do something
like javac, client, or java.
But just inspecting the files in the command itself
does not give us a complete idea of the program.
We have no idea to figure out that Lib.java is also included
and we have no idea to figure out,
no way for us to figure out what was in Lib.java.
So we attach a Dtrace process to the javac shim
and essentially what it does is,
it reads every file that was written by the javac shim.
And this lets the creator of the tutorials
see every file that was ever compiled.
And again this is a very trivial example,
this also allowed us to see things like
configuration example files, and data adjacent files
and a lot of other things.
So once we have all of this data,
we obviously want to visualize this,
there is no point of having the data
if you can't see it, right?
We want to answer questions like,
where in the work page should a particular event happen?
We want to answer things like,
when did this event, when did a particular event happen?
And also we wanted to be able to filter events
based on things like, between certain time periods
or between certain sub-sections of the tutorial
or to filter events by their types.
So this was the UI that we envisioned for visualization.
We see the original tutorial work page.
On the left we see a positional heatmap
for the user's focus area that we discussed earlier.
On the bottom we see a temporal heatmap
for the events that are based on time.
The event icons are laid out both on the positional heatmap
to denote where on the page they occurred
and on the temporal heatmap to show
when in the tutorial they occurred.
And on the right you can,
the creator can filter events by the event type,
and there are also temporal and positional range sliders
to filter events based on time,
or to particular sub-sections of the tutorial.
And this is kind of how it looked when it was implemented.
So when the creator clicked on a particular event icon,
he got a pop up with more details about the event.
For example, in this case, you see
the working directory and the return code, etc.
So now that we have this system built,
we want to see if Porta can,
firstly enable learners to reflect of the difficulties
they faced while following tutorials,
and also to see if instructors could
gather useful feedback to improve their tutorials.
So, for the evaluation, we used real world tutorials.
The first one was a Python tutorial which is a primer
on types, control flow, and functions.
It was using two iterations of
a 400-student data science course.
The second was a web design course,
which was an introduction to HTML, CSS and JavaScript,
which was using four iterations of a 300-student HCI course.
The last was an introduction to Git and GitHub
which was using a 10,000-student online course.
So we recruited four undergrad learners for each tutorial,
which is about twelve learners in total,
nine female and three male for about one hour each.
All of them had little to no experience
in the subject of the tutorial.
So first the four learners look at one particular tutorial.
They then reflect on the difficulties and improvements
that they can make to the tutorial,
initially, before seeing their own Porta visualizations,
and afterwards while seeing their Porta visualizations.
We then had the instructors look at the recordings
for those four learners.
And then the creators reflected on the improvements
that they would make to the tutorial.
Firstly, before seeing the four recordings,
and after a while seeing the four recordings.
So this was an overview with all three tutorials.
And for the learners, the learners actually had feedback
for after they saw their Porta output.
So eight of twelve went from something like,
"It's good enough in its current state,"
to having actual feedback.
The learners gave very specific feedback.
They went from something like,
"The language could be more novice friendly,"
to, "Use different formatting for text and commands."
Or something like, "Need more Windows-specific advice
to explain how the cat command was used here."
The learners also gave targeted feedback about
where in the tutorial page they wanted improvements.
So for example, one of the learners mentioned that
that the tutorial ought to mention that
the variable in a Python for loop is the value in the loop,
and not the index, like in java.
And for the instructors, the instructors were able to
overcome their own expert blind spots.
All three instructors noticed unexpected
student behaviors that surprised them
and this was despite these tutorials
having been used multiple times in the past.
The instructors also came up with
very specific improvements.
The instructors went from, you know,
something like "Update this tutorial for Python 3,"
to "Talk about starting code blocks
with colons and indentation in Python."
So this is a high level overview of the impact of the study.
With just about twelve learners,
three instructors, and about five days,
we were able to solicit 28 unique improvements
from the learners, as denoted by
the pink arrows in these three images,
and about 27 improvements by the instructors,
shown by the green arrows in these three images.
And again, just to reiterate, this was on tutorials
that had been use extensively in the past
and had been iterated multiple times.
So to summarize, we built Porta, which is a system
to record learners as they used tutorials across
multiple applications, include IDEs,
terminals, text inserters etc.
And then visualize all this data
on top of the original work page.
The learners were able to better reflect
on their struggle while using this tutorials.
And the instructors were able to gather
more actionable feedback for the tutorials.
I'd love to answer any questions that you guys have today.
(clapping)
- [Announcer] Please step up to the mic with your question.
- [Woman] Hi, I have two questions,
first I... verifing the layers, I don't know
if you'd talk about the method.
How did you validate that using the mouse
and the scrolling is actually showing user focus,
because focus is also different than attention?
And also, did you run a study using eye trackers
to see the comparison of the results
that you can get with eye tracker
when you measure gaze and when you're selecting
this method that you selected for the particular study.
- To answer the question, no it was not a very formal study
in terms of the accuracy of the eye tracking.
But what we saw in our experience was that
the positions where people actually spent time
was where it was highlighted in the tutorial.
Where they were actually looking at
that was where it was located, yes.
- [Woman] So you didn't even ask the users if they were...
- Oh yeah and the learners also agreed that this was
actually where they were looking at.
But it was not a very formal study.
- [Woman] Okay, thank you.
- Thank you.
- [Man] Hi, while the focus of this work is
for improving the tutorials, was there any case or examples
when watching was learning praise again,
helped...
Students'...
learning?
- Oh that is an interesting question.
We did not try that out, but that would be
an interesting future work, to see if
looking at their profile actually
helped them learn better. Thank you.
- [Man] Thank you.
- [Carson] Hey hello Carson Zior from Google.
Thank you very much,
this is a very interesting presentation. I've two questions.
One is, did you run the follow up study to see
about the efficacy of the proposed changes?
So you say, "Hey you improved
stickiness of learning by, say, 30%."
And the other is, do you have any funnel diagrams or so
that you could deduct from the typical path
of doing a tutorial?
For example, do you see everybody starts in a browser
then to the IDE then twice back overflow
then back to the browser, something like that?
- To answer your second question, I'm sure we should
be able to visualize that from the data that we have
but we have not done something in that regard so far.
Could you repeat the first question again?
- [Carsen] I think the first was about the efficiency
of the improvements that were made
with regards to the feedback that was gotten
by tracking the ideal user flow.
- Yes, so we actually did have a study on that,
which I did for another course while I was at UCSD.
The improvements that were made were actually impactful
in terms of, it did improve what the learners
thought they saw initially.
But unfortunately we could not include in this paper
because it was past the time, and I did this
for a different course.
- [Carsen] I totally understand,
but I think what would be super interesting to see
that there was any quantifiable data,
to say, "Hey, you know, people now prefer 20% less time and
are 10% more knowledgeable about whatever has to be taught."
- [Carsen] Thank you very much, cheers.
- Thank you for the great questions.
- [Announcer] All right, lets thank the speaker.
Congratulations.
(clapping)
So this concludes the session on
things that are not AR, VR or fabrication.
Thank you for joining.