- Thanks for the introduction, my name is Volodymyr,
full name, people call me Vlad,
and I'm a PhD student at the University of Manitoba
and today I'm here to talk about the work I did as an intern
at Autodesk Research one of the previous summers.
It's called Maestro: Designing a System
for Real-Time Orchestration of 3D Modeling Workshops.
(clears throat)
So children often learn 3D modeling in group workshops
and they are often structured in the way
that they have an instructor who is guiding the children
through a step-by-step tutorial
and children just repeat their instructions step-by-step.
However, prior work tells us that
facilitating these things as an instructor
is very challenging because, first of all,
you need to be aware of what a student is doing
so that you know who is falling behind,
who is following the instructions,
and when it's safe to move to the next step.
You also need to provide assistance
to those who struggle, who cannot keep up
with the pace or have problems,
and also you need to manage all the social interaction
that happens between kids and between yourself
as an instructor, and these are very generic challenges
that can apply to any type of workshop,
but they're also domain-specific challenges.
For example, in 3D modeling workshops,
it's very common for children who just started
to develop something from the same angle,
same perspective, and never change their view,
because they're not used to being in 3D,
and this results in their models seeming to be okay
but in fact they're not really connected at all,
which could be a problem when you try to 3D print them.
So in this work we present Maestro,
which is a system that does some real-time analysis
of student command data to help instructors facilitate
these workshops by detecting common challenges.
There are other systems in both research and on the market
that look into this one-to-many tutoring assistance.
For example, the system that is
on the market is NetSupport School.
It's a system that allows you as an instructor
to see a view of each student's screen
and to do some generic interventions such as,
for example, taking control of a user's input.
Our approach dives into the depth
of a specific domain, in our case 3D modeling,
and this allows us to use domain-specific knowledge
to help instructors detect potential problems
that are common in this domain
early in the stage of their development.
There's also been research on one-to-many
remote tutor interfaces, and probably many of you know
the system called Codeopticon,
and our approach is a bit different
because we focus on in-person types of workshops,
where there's lots of social interaction,
and we aim to augment this interaction,
to extend the instructor's reach,
while not replacing this interaction with a system.
We also work a different domain, because,
in 3D modeling, we don't really have an objective identifier
of an error, such as a compilation error.
Overall, we do the following contributions.
First we present an idea of data-driven real-time assistance
for conducting in-person 3D modeling workshops,
wow that's a mouthful,
we also present design and implementation of Maestro,
and finally we present an evaluation of Maestro
in a workshop with an instructor and 13 children.
So without further ado, let's jump into the system.
So Maestro works with Tinkercad.
Tinkercad, for those who don't know,
is an online tool that allows you
to do some basic 3D modeling, and it's very accessible
and it's open, you can join, you can try it for free,
and that makes it very popular for these kinds of workshops
to introduce children into 3D modeling.
And when designing Maestro, we consulted
prior work, and we also consulted two
experienced workshop facilitators from local makerspaces,
and we arrived at the following set of design goals.
First, we wanted to provide a glanceable overview
of students' progress.
We also wanted to draw the instructors' attention
to relevant activity patterns.
We know that, from prior work and from our interviews,
that these patterns might indicate
potential problems or good practice.
We also wanted to augment instructors' capabilities
with quick interventions, so they can serve their students,
and last but not least we wanted
to leave the initiative to the instructor,
and make the instructor the one who decides
what to do, what interventions to set, and (unintelligible).
So the way Maestro works is it looks at the command logs
from students in real-time, and it displays
for the instructor a dashboard,
where each student is represented by a widget,
and on this widget, the instructor sees a live view
of the student's workspace, and also real-time
command activity from the student.
So after consulting prior work and the workshop facilitators
we decided to split the whole command activity
into three timelines.
The top one shows the object modification commands,
the middle one shows 3D navigation commands,
and the bottom shows the undo and erase commands.
Further, Maestro also uses heuristics
to try to analyze these logs in real-time,
and it looks for activity patterns
which we know from prior work and from interviews,
these activity patterns indicate
either good practice or potential problems.
And then Maestro recommends the instructor
to send quick interventions based on these heuristics
that could either encourage the student for doing a good job
or they could help the instructor prevent a potential error.
And let me just guide you through
a couple of examples of these heuristics.
First is the 3D navigation reminder.
As I mentioned before, it's very common
for children in these workshops to never change their view
and do everything from the same angle, which causes problems
and what Maestro does is, when it detects
that a student has been working for a while
without any navigation, it highlights it on the dashboard,
on the command log, and also it recommends
to the instructor an intervention,
and if an instructor accepts this intervention
by clicking on it, it shows a message
on the student's screen reminding the student
to look at their model from different angles
to make sure that everything's okay.
Another example of these heuristics
is follow-along detection.
As I mentioned before, these workshops
are often structured in a follow-along way,
where you have a tutorial and the instructor's going
step-by-step, waiting for students
to complete the previous step before moving to the next one,
and what Maestro does is, when it detects
that more than half of a class
repeated the command after the instructor,
it switches to a follow-along mode,
and in this follow-along mode, the instructor
sees a real-time indication, in a very glanceable way,
of who has followed the command and who has not.
More than that, Maestro also keeps track
of who is following the instructions well
and recommends to the instructor to send these students
a thumbs-up intervention, and this is meant
to encourage students, and the way it shows up
is as a flashing thumbs-up message onscreen,
like "Yay, you did well."
In addition to these heuristics,
Maestro also allows the instructor
to send manual interventions to students,
such as the ability to point to a location
on a student's screen, and also to ask one student
to go and help another student.
We evaluated Maestro in a workshop
with 13 children and an experienced workshop facilitator,
that we hired from a local makerspace,
and our goal for this evaluation was twofold.
First we wanted to get feedback
on this whole idea of augmenting
in-person workshops with data-driven assistance,
and also we wanted to collect data to check our heuristics,
to see that they were performing
what we intended them to do,
and the first type of data we collected was command activity
so this image depicts all the command activations
that happened during the workshop
from instructor to students, and the way we used
this very colorful and nice graph
is to check our heuristics, as I mentioned.
For example, the way we checked it is,
let's talk about the follow-along heuristic,
it's probably the most complex one that we had.
The way it works again is it activated
the follow-along mode if at least half of the classroom
followed the instruction from the facilitator,
and inspecting this heuristic against the data
showed that it was quite successful,
it detected many scenarios from
when students were supposed to follow the instructor,
but there were also some cases
where it did not detect that,
and that was something we were looking for,
and one of these cases - this is the image.
So this is happened for the 'Create Torus' command,
and here you can see a zoom-in
on a specific area through the entire workshop,
where you see a cluster of activations of this command,
and here you can see also the instructor's command,
the instructor's demonstration is marked
by the vertical line, and all the other dots
represent the students' invocations of this command,
and we saw that our heuristic did not work here
because most of the students activated the command
before the instructor, and we were curious
about what happened and we looked into the video
of the workshop, and we saw that this happened
because the instructor first asked the students verbally
to complete the command, and after 10 seconds
she repeated it herself, and of course
this suggests a direct improvement
for heuristics to also include
those command activations from students that happened
before the instructor's demonstration.
In addition to collecting and looking into log data
we also conducted an interview with the instructor
and in this interview we asked her
to compare her experience of running this workshop
with Maestro to her prior experiences
of running the same kinds of workshops without the system,
and one of the things she mentioned was
that follow-along detection helped her to pace the workshop.
Specifically, she said that: "I really liked the function
"where I could see if they followed my direction.
"I thought that was great, the screens lighting up green
"and red, it just draws my attention
"to a really quick glimpse sort of way
"who's with me and who is not.
"I don't have to rely on the students
"to let me know if they did not get the step,
"and then I won't speed ahead to the next thing."
So in this quote she referred to the first half
of the workshop, where she was doing
a tutorial with students,
and this quote in our observation suggests
that having this glanceable, real-time feedback
could really increase the instructor's confidence
in acknowledging that it's safe to move to the next step
and not to leave students behind.
Another piece of feedback we received is she found
that Maestro extended her reach to the students.
Specifically, she said that: "Instead of me being like
"'just a second' and coming over to the other student,
"I could just use point-to intervention
"and the student would see the command's location
"on their screen.
"This way I can stay with my really high-need kids
"and do my quick interventions
"with the system for common problems."
This finding suggests that there is room for improvement
in helping the instructor to reach more students,
especially when they have these very common,
easy-to-fix problems such as not being able
to find a command on the interface.
And the same point is also demonstrated
by a different quote from a different angle,
and here the instructor said that:
"A lot of times students think they followed the instruction
"but they are not sure if they did.
"So giving them like a quick-thumbs up
"like 'you got it,' is good, and then they know
"that they could wait for the next instruction
"if they don't know if they got it right.
"I just see it as a potential way to say
"'You are good, wait.'"
This quote suggests that there's also room for improvement
in terms of helping the instructor
to provide feedback to students on their work.
And together these findings suggest the value
of these quick and pre-authored interventions
would be for giving feedback to students
or would be for helping them with potential problems.
Another interesting finding was
that the instructor felt that Maestro
gave her objective data that could help her
avoid biased views on her students.
Specifically, she said that: "It is very important
"for us teachers to be able to see
"where our students' strengths are
"and a lot of the time it's hard to see it
"because we have jaded ideas about certain students
"if they caused us a lot of grief or whatever."
This quote suggests that this objective data
could help us remove some of this bias
from the judgment of the student
from the instructor's perspective.
Finally, we also found that there could be big challenges
with splitting attention during these workshops.
Specifically, the instructor said that:
"Paying attention to both the classroom
"and the tool was really hard.
"I felt really torn and my inclination was
"just pay attention to the children and the classroom,
"but I also want to use the tool because I see its value."
This suggests that high cognitive load
could be one of the big challenges
for this kind of real-time in-person workshop assistance,
and that future work should investigate the ways
we can reduce this cognitive load.
Overall, our findings suggest that the approach
of using real data and analyzing it in real-time
for assisting in-person workshops is quite promising,
and we also uncovered some directions for future work.
For example, one direction is using
more advanced learning modeling techniques,
because our approach used quite basic heuristics
which were based on the command sequences,
which worked pretty good, but one can imagine
that a richer model of the student
could help make more accurate predictions
and more accurate suggestions,
and this more accurate suggestion mechanism
could actually reduce the cognitive load
which I have been talking about.
Another interesting direction to explore
is allowing students to interact with systems like Maestro,
because one can imagine that it could be interesting
to see what happens if students also
can use this kind of system to ask the instructor
for help or for feedback directly.
And finally it would be really interesting to investigate
how this idea generalizes to a larger context,
because you can imagine, if the workshop is quite big,
there could be more than one instructor
or there could be an instructor and multiple TAs
and it would be really interesting
to see how this would scale.
That's it, thank you for your attention
and let me know if there are questions.
(applause)
- [Usher] So while people are thinking of asking a question
- Oh, okay we have one person.
- [Audience Member] I'm Takeo from University of Tokyo.
Do you support interaction between students?
- We didn't really support it through this system
we did something to ask the instructor
to ask one student to go help another student,
so we've built this kind of intervention in there,
but it wasn't honestly used in the workshop
and when we asked the instructor why,
she told us that she still prefers
to do these things in person, and also
in this workshop it didn't really make sense for it
because it was her first time with this audience
and she told us that if she were more familiar
with those students, she would know who to send to whom,
who is the fast learner, who can be struggling,
and maybe in future stages of this
when she's more familiar with the classroom
she could use that feature more,
but we didn't observe that use right now.
- [Audience Member] So I have one question.
I know that you've been thinking
about supporting new learning styles
and perhaps new classroom environments.
What are your thoughts in terms of extending the system
to support collaborative projects
where you might have groups of students
learning or working on projects,
some of them might be in the classroom,
some of them might be remote.
- Do you mean collaborative projects
when all students work on the same problem?
Yeah this could be a very interesting interaction
to explore, the initial assumption that
if you just track the number of machines, let's say,
it will just decrease out of the classroom,
but if we want to look past that,
for example, you can also look
at the contribution of each student into the project.
That could get very tricky (laughs)
but that's a very interesting direction to explore.