Hi! I'm Meredith Young-Ng and I'll be presenting Portalware
Exploring Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable Paradigm.
This is work done with Jing Qian, Tongyu Zhou, Jiaju Ma, Angel Cheung, Xiangyu Li, Ian Gonsher, and Jeff Huang at Brown University.
Traditional drawing flattens the 3-dimensional world onto a 2-dimensional canvas.
Although we have used techniques such as linear perspective and foreshortening to create the illusion of a 3rd dimension,
sketching in 3D was still impossible since we were constrained to a flat canvas.
However, augmented reality systems have enabled us to finally overlay virtual ink onto a physical 3D environment.
AR drawing on smartphones allows us to illustrate a 3D scene on a 3D canvas, allowing sketching to occur nowhere and everywhere.
Prior work used head-mounted displays, handheld devices, and wearables to explore free-hand sketching in AR.
Head-mounted displays provide accuracy and immersion for sketching in AR, but lack ubiquitous mobility and thus practicality in everyday scenarios.
In contrast, both handheld devices and wearables offer wide availability and portability for everyday use cases,
but lack significant exploration with free-hand sketching.
Therefore, little is known about what benefits and design and interaction challenges users face when using either a smartphone or a wearable device for free-hand sketching.
Given that smartphones and wearable devices are promising candidates for freehand mobile AR sketching, what interaction paradigms need to be explored?
We engage in two-stage explorations: single screen vs dual screen.
We will first discuss the Portalware smartphone single-screen format.
In Portalware’s smartphone interface, users stylize stroke color, width, brush type, and toggle between the 3D and 2D user interface.
When sketching with a single screen, the 3D color picker is toggled by holding the 3D button (typically) with the left thumb.
The location of the drawing point (a light blue sphere) is highlighted on drawn strokes to indicate depth.
Now, we will discuss the Portalware smartphone-wearable dual-screen format.
This is an example of a user using Portalware to draw a 3D plant in a vase.
The main hardware components of the Portalware dual-display system are a Raspberry Pi 4 and its 3D-printed case,
a SSD1351 OLED wearable display, a Samsung Galaxy S10 smartphone, and a dual-fisheye depth camera for hand tracking.
In our system pipeline, a Websocket server transmits data from the depth camera to the phone,
which determines the background of a virtual camera at the same position as the wearable and then transmits it to the OLED display.
After testing different positions, we mounted the wearable on the fingertip.
Transformations of the palm are applied to the virtual camera, and the finger's position and orientation determine the rendered AR background.
Rapid hand rotations during sketching can cause the virtual camera to move beyond the AR background texture.
We limit the virtual camera’s rotation by using a remapping method applied to a Kalman filter.
Unconstrained, the virtual camera rotates excessively when the user rapidly moves their hand.
Upon applying constraints, it becomes more stable.
Finally, we will discuss our autobiographical design method.
Our goal was to elicit design lessons for mobile AR sketching by following Neustadter's 5 autobiographical design tenants
(genuine needs, real systems, fast tinkering, record keeping and data collection, and long-term use).
The participants were 3 author-users with both an art/design and a computer science background,
focusing on visual perspective, ergonomics, and hand-eye coordination.
The procedure involved documentation via screenshots, videos, and written notes over a timeline of 14 weeks,
including the iterative process. The tasks involved were a single-display, then dual-display smartphone format.
These are examples of sketches from the smartphone single display format.
The author-users found that the ability to hide previous strokes was helpful in drawing 3D objects,
and that the 3D color picker was intuitive to learn and use because its interaction pattern was similar to drawing.
They noted that the experience was like drawing on an infinite canvas, and that sketching is much easier in 2D.
They also found that when they reached beyond the smartphone’s view, they were unsure whether a stroke was made,
and that they couldn’t tell whether new strokes were connected to previous ones.
First, limited screen size restricts visible free-hand movement.
As shown on the left, the smartphone screen can only show a limited range of hand movement,
but our hand has a much larger ergonomic hand movement range as shown on the right.
Next, users experience a loss of spatial awareness when sketching on the plane perpendicular to the screen.
Users are unable to see this dimension through the smartphone screen, as shown in this figure.
In addition, creating planar drawings is faster and easier than creating well-formed 3D drawings with mobile AR.
Planar 2D drawings are completely visible via the smartphone screen as shown on the left,
while the smartphone must be moved to different angles to view the different perspectives needed to create 3D drawings as shown on the right.
Also, 3D user interfaces feature improved ergonomics compared to 2D interfaces.
The gesturing hand can easily maneuver the 3D user interface in AR space as shown on the left,
but is constrained to screen space when using a 2D user interface as shown on the right.
Finally, users create their own personalized workflow to solve drawing challenges.
First, "to smooth the drawing process, I gradually developed a steady workflow loop of sketching for 1 minute then resting for 5 seconds on repeat,
and was able to create more complicated drawings through this method.”
Next, “for multi-planar (3D) drawings, I first established a center point, then revolved the phone hand around that point.”
And finally, "for 3D drawings, I drew strokes to define the boundaries before filling in the boundaries with repeated parallel strokes.”
Here are some examples of sketches from the smartphone-wearable dual display format.
For the dual-display format, the author-users found that they felt that the sketching canvas was larger and that drawing time for 3D objects like cubes decreased with the wearable.
The wearable provided a zoomed-in view and made it easier to connect strokes,
and they found that they could more easily draw circles with the phone’s height as the radius because of the extended field-of-view.
For 2D drawings, the wearable did not improve drawing and was at times distracting and superfluous.
They couldn’t detect color [depth] changes when moving the drawing hand in and out of smartphone view.
And finally, it was still hard for them to determine how far away they could move their hand before experiencing hand-tracking issues.
Based on analyzing the dual display results, we developed the following design lessons for dual-display AR sketching.
A wearable display is helpful for drawing 3D (multi-planar) objects and expanding the "perceptual canvas."
In this clip, the user demonstrates the perceptual canvas by drawing two primitive shapes: a cube and a cylinder.
Next, wearable depth indicators are helpful for determining tracking range.
In this figure, the wearable displays a blue color when the hand is outside the tracking range and a yellow color when the hand is inside the tracking range.
Finally, a wearable display can help reduce the cognitive gap between visual input and output positions.
Without the wearable, as shown on the left, we have to mentally compute the cognitive gap between the input position of the fingertip
and the output position displayed on the smartphone screen.
With the wearable, as shown on the right, the wearable can help reduce the cognitive gap by displaying the output position on its fingertip-mounted display.
Ultimately, the autobiographical design process lacks users from different skill levels. We are thus unable to extend these insights into formal guidelines.
Future work will focus on enhancing the generalizability of design lessons.
These include recording quantifiable metrics through human subjects experiments,
conducting supplementary testing with mixed methods such as usability testing,
and evaluating how learnable the interaction techniques are with unfamiliar users.
In summary, Portalware is a working system that enables free-hand AR sketching for both a single-display and dual-display format
with an autobiographical design study motivated by genuine needs.
It features design insights that explore the perceptual, workflow, ergonomic, and interaction challenges for free-hand sketching.
Thanks for listening! Please reach out with any questions.