So let's start, hello, my name is Germán and I will be presenting Pronto.
I hope you are seeing the presentation.
So, let's start by saying that prototyping is at the core of interaction design; from
static sketches in a napkin to quickly visualise the user flow, to physical prototyping with
clay, for example, to explore the form of a design.
However, some designers rely on video to prototype dynamic systems' responses (and we saw it
recently) *reference to a previous presentation*.
They combine paper prototyping with the Wizard of Oz technique to let them play the role
of the computer.
But, how a technique like video prototyping can be used specifically for Augmented Reality?
Well, if we have time and money we can use video editing software to create an illustration
of the AR experience, like Google did here with Project Glass.
If not, in 2018, I introduced a new video prototyping tool that is called Montage, that lets designers
quickly sketch and animate on top of video.
However, this is only in 2D, faking the AR experience without any 3D information.
So what are the current tools that give AR designers access to 3D positioning in space.
First, we have the development tools, they require programming and include game engines
such as Unity or Unreal, and we have also SDKs like ARKit for iOS and ARCore for Android,
but these slowdown the prototyping speed, particularly in early-stage design, and also
exclude non-programmers.
There are few commercial prototyping tools that don't require programming, however, they focus heavily
on face tracking AR or they provide a limited set of pre-created interaction triggers.
To sum up, current tools generally focus on pre-made assets instead of creation, static
or fixed animations rather than dynamic scenarios, high instead of low visual precision, and
default interactions rather than exploratory user inputs.
With this in mind, what would be the design goals for an early-stage AR prototyping tool
that is simple to use but also powerful enough to create open-ended designs?
To inform the design of this tool, we conducted three formative interviews with expert AR
designers to identify current issues in early-stage AR design.
Four problems stood out.
First, all participants mentioned a vast ecosystem of tools to create 3D assets.
Other problems are a consequence of the freedom that the user has while using the system.
One of the biggest challenges when designing for AR is the designer's inability to control
the camera.
Participants also said that the most tedious task is positioning 3D assets.
P2 mentioned that positioning is problematic with depth and occlusion.
Also, designers consider animation a pain point, they avoided laborious manual animation,
like key framing, during early-stage prototyping.
To tackle these problems we propose the following goals, one extra implicit goal is providing
a lower barrier of entry to any stakeholder.
To achieve this, we made the following decisions.
We will use a single mobile device to provide a low barrier of entry.
We will use sketching to create low fidelity assets.
We will use video capture to capture the always changing position of the user.
And direct manipulation, again, through demonstration to position assets and to animate them.
With these decisions in mind we built Pronto.
Let's introduce Pronto with a scenario. Imagine a designer that is prototyping an augmented
reality smart-home.
He wants to place AR objects on top of a coffee table.
Turn on the TV with a flick of a finger.
And send likes with mid-air gestures.
The first step is "acting and capturing", this happens in Pronto's RecorderView.
After a 3 seconds countdown the designer starts moving in the environment as if the AR system
is already in place.
Sorry, let's put that again.
He moves around, he sees something on the coffee table, then he looks at the TV, he
swipes his finger to turn it on and then he uses a pinch-and-push gesture to send a like.
Step 2 is "creating sketches" in Pronto's PlaybackView.
The previous video is loaded and now the designer can use the tablet in the physical space to
position objects in the current frame of the video.
This happens in real-time, the position in the physical space is represented in the current
video frame.
The tablet works as an avatar to position "sketch layers".
For example here, the designer decides to put a new sketch on top of the coffee table
and creates it.
After that, he can change the size and, in this case, he will draw a weather widget showing
that is sunny.
The asset remains in place during playback.
Step 3: "enacting an animation".
The designer adjusts the video and moves towards the TV to create a sketch in the TV's screen.
Then he draws the content in the TV by creating a new sketch layer, resizing it, and, in this
case, he will draw a cute cat.
Now, the cat needs to be hidden and then be revealed with a flick gesture.
The designer manipulates the top handle and pushes it down.
He can use the "enact" button on the left to start recording and demonstrating the system's
response.
In this case, he is pulling the same top handle but going up.
He presses the "enact" button, and does the animation.
The animation is saved in the timeline, and its duration can be adjusted to match perfectly
the user input.
For the like interaction, he first needs to create a heart at the pinch position, the
designer wants the heart to fly towards the TV with a custom spiral animation.
Now, instead of manipulating handles, we will move the whole sketch, the heart, with the
tablet position as a proxy.
This is what we call "direct enaction".
The designers presses "enact", and then moves towards the TV with a spiral path.
The animation is saved and he can adjust the duration in the timeline.
Finally, to make the cat react, he uses the same process with a shake motion.
And this is the final result made in less than 4 minutes.
We can see the weather widget on top of the coffee table, how the user turns on the TV,
and sending a like with the response.
In terms of related work, there are many AR prototyping tools but they require extensive
programming.
However, there are a few, general non-specialised prototyping tools such as DART.
However, it still requires extensive coding skills to customise the experience.
Other tools combine AR and Wizard of Oz but without the video capturing capabilities,
making it difficult to synchronise (wizards and interactions).
For example, 360Proto specialises in 360 experiences by creating paper prototypes.
ProtoAR combines paper sketches with clay to illustrate quasi 3D effects.
In contrast, Pronto focuses on dynamic interactions.
And PintAR combines digital sketches, in a tablet, with live AR tracking.
It does not support the creation of custom animations and uses video just as an output
and not as a design medium.
In terms of the implementation, we have these 2 views.
The Recorder and the Playback.
The RecorderView is a typical AR session capturing video frames, fixing the world's origin, tracking
the position of the tablet, and also the position of assets.
This information is saved during recording in 2 files: one that has the timestamps and
transformations, and a movie file with the pixel buffers.
Then we recreate the 3D scene in the PlaybackView but without the live video feed we use the
video file as the background of the camera, and that will mimic the 3D scene and it is
represented in the player.
To evaluate Pronto we conducted an expert review with 6 AR designers.
The first task was to recreate the Pokemon GO capturing interaction with mid-air gestures.
All the participants were able to finish this task in less than 20 minutes and it took
12 minutes in average.
Then task 2 was open-ended.
For example, P1 created a narrative-based experience.
He created an off-screen attention guiding cue and then this little animation with the bee.
P2 created a racing game with some difficulties.
So let's move on.
P5 for example, did a smart-home experience were each bin knows the type of garbage that
it can receive;
and P4 created a product tutorial with instructions to operate this tumbler.
Basically, participants successfully recreated these advances AR mechanics using Pronto in
a short amount of time, the average completion time was 13 minutes and 3 seconds, and all
participants ranked high Pronto's capabilities for exploration and creativity.
In terms of future work, we would like to suggest several directions: one is adding
collaboration to support multiple designers, to add comments and annotations, and also
to take consideration depth to fix occlusion problems, and to move from video illustrations
towards interactive prototypes. Thank you