Hello! My name is Shwetha Rajaram and I will be presenting Paper Trail:
An Immersive Authoring System for Augmented Reality Instructional Experiences.
This is work done at the University of Michigan along with my advisor, Michael Nebeling.
Prior work has established augmented reality as a really promising modality for supporting
educational activities, but we can observe two challenges with current AR authoring tools.
First, teachers may still rely on physical instructional materials especially involving
paper, like paper handouts or lab notebooks, and augmented reality can be really difficult
to integrate with these existing instructional methods. Additionally AR authoring tools are
often geared towards designers and developers and not specifically towards instructors. So,
to explore how instructors could complement their existing teaching workflows using AR,
we designed and developed Paper Trail, an immersive authoring system which allows
instructors to extend their paper-based teaching resources with various kinds of AR media to
support learning. I will first present our design process and then discuss one of our evaluations,
an elicitation study with experienced high school instructors. To inform the design of Paper Trail,
we first adopted a scenario-based design approach by creating five target use cases to explore how
AR could enhance paper-based activities in a variety of science and technology domains.
We implemented every scenario on either the iPad Pror or the HoloLens 2, so that we could explore
benefits and limitations of both hand-held and head-worn AR. This immersive chemistry lab
notebook allows students to integrate a variety of digital media with paper using head-worn AR.
They can insert audio clips to record observations while performing an experiment, annotate digital
images, and refer to a video for the procedure. In our Biology Poster example, an instructor extends
an existing poster of a cell using clipping masks to selectively hide and show content. Students
can expand the clipping masks to zoom into organelles and view details about their functions.
To further explore potential interactions to adapt for Paper Trail, we developed a
design space of immersive paper interactions for instructional activities by synthesizing prior
work on interactive paper and AR authoring tools. We first categorize these existing
systems according to common tasks they enable, and then we mapped these tasks on a spectrum
according to whether they're mostly grounded in the physical space or the digital space.
The first class of interactions leverage paper as a tangible user interface for manipulating
physical or digital content, including detecting paper-based gestures and embedding physical
components like paper-based electronics, to provide users with visual or auditory
feedback. The second class is capturing and embedding digital content in paper,
including capturing handwritten content using digital pens, creating digital media from paper
using camera based techniques, and associating digital content like photos and videos.
The third class involves transforming paper-based digital content, including authoring techniques
to move, scale, and rotate content, showing and hiding digital content, and animating content.
And the fourth class of interactions involves converting physical content into purely digital
formats, including generating digital documents or 3D scenes from handwritten or hand-drawn content.
We found that interactions in the center of the spectrum were most feasible to
adapt to instructional use cases, while preserving the physical benefits of paper.
After extracting system requirements based on our target use cases and design space,
we developed a new immersive authoring tool: Paper Trail. We adapted some of the introductions
from prior work, including techniques to capture digital content, performance animations, clipping
masks, audio and video clips, and shared content. And to study the affordances of both hand-held and
head-worn AR in educational contexts, we developed interfaces for both the iPad Pro and the HoloLens
2. Now I will walk through the example of an AR-enhanced instructional handout for torque
and angular momentum, which we created using Paper Trail and utilized in our user studies.
First, to demonstrate the direction of torque in a printed diagram, a physics
instructor captures a hand-drawn vector as an AR object and aligns it popping out of the paper.
Next, she uses animations to create dynamic illustrations of two scenarios where a
force is applied to a uniform rod. To provide students with hints on a derivation problem,
the instructor integrates an audio recording and digital post-its using clipping masks. Students
can expand the clipping masks to reveal the hints. Finally, the instructor captures a video of a
gyroscope demonstration, to provide students with background information within the handout itself.
Our paper discusses two evaluations with PhD students and XR design experts to refine the
design of Paper Trail, but in this presentation, I will just focus on our elicitation study with high
school instructors. So, to explore whether and how instructors could envision using AR to complement
their existing teaching workflows, we conducted an elicitation study with seven high school
instructors who had an average teaching experience of 18 years. In our study, they explored how AR
could enhance a variety of topics that they currently teach, including physics and math,
sustainable energy, computer aided manufacturing, cellular biology, and media and research skills.
The study consisted of three tasks. First, a walkthrough of a paper-based teaching resource
which the teachers brought with them to the study. Next, a review of our AR torque handout using both
the iPad Pro and the HoloLens 2. And both of these tasks were used as priming for the third task:
an elicitation task where the instructors proposed and prototyped AR interactions for
their paper-based teaching resource, using either the iPad Pro or the HoloLens 2. The instructors
prototyped a wide range of interactions to enhance their own teaching materials. A physics instructor
prototyped a problem set using AR to simulate how a roller coaster cart is affected by different
forces, like spring force and friction. Another instructor adopted a macromolecule building
activity where students can directly manipulate and break off molecules to form new carbohydrates.
We found that a majority of proposals involve transforming or associating digital content
with paper. And, a majority of instructors chose to prototype using hand-held AR,
which they found more practical and efficient for authoring.
However they thought that head-worn AR would actually be a more natural form factor for viewing
immersive paper experiences, especially for lab-based activities or collaborative teamwork.
In terms of opportunities or benefits that immersive paper could provide to educational
contexts, instructors expressed that it could be a really promising modality for supporting students
diverse learning needs. They could engage students with multiple visual and auditory representations
of course material and allow them to demonstrate their understanding in multiple ways. Instructors
also felt that immersive paper can support adapting established pedagogical models to an AR
format. As one example from our study students could first make predictions about a physics
scenario on paper and then they could use AR animations to simulate the scenario. Although the
instructors could brainstorm and propose many ways that AR could complement their existing workflows,
they still had some open questions about what is the best way to translate students learning goals
into AR interactions. And this raises a need for more concrete design guidelines which take into
account how to also preserve analog instruction and learning strategies while introducing AR
to paper. And they also noted that there is still a technical barrier to entry with AR,
and from their past experiences introducing other kinds of digital technologies in the classroom,
the instructors felt that there is always a need to provide meaningful alternatives for students
who are not comfortable with technology or who may experience accessibility issues as well.
Thanks so much for watching our presentation, and please refer to
our paper for more details on our research process and findings from our user studies.