Hello everybody and thanks for watching my talk. My name is Tobias Drey. I'm from
Ulm University and I want to present you VRSketchIn: Exploring the Design
Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. First of
all I want to jump right into it and show you our system. Here you can see an
in-game capture and on the bottom left you see the person who is using the
system. VRSketchIn is a VR sketching application and it uses six
degrees of freedom tracked pen and six degrees of freedom tracked tablet as
input devices. This combines unconstrained 3D mid-air sketching with
constrained 2D surface-based sketching. To explore what possibilities
arise from this combination of 2D and 3D input, we present a set of design
dimensions and span the resulting design space for the sketching
interaction metaphors. But why did we did all this? Sketching is an important part
of the design process. It enhances visual thinking. Sketching in VR improves the
understanding of 3D volumes but it's challenging as in mid-air there are no
constraints and the controllers are not made for sketching, they are optimized
for gaming. Here you can see Glen Keane. He's a very famous designer from Disney
and he created popular movie characters like Aladdin or Ariel. In this video you
see how he's using a VR sketching software which is focusing on the
unconstrained 3D mid-air drawing part. This has all the previously explained
challenges.
When you look at how he's working nowadays, then you see that he is still using pen
and paper, and he puts them on his desk, which gives him all the benefits from
surface supported drawing. We want to merge the best of both worlds and create
a system for interchangeable and unobtrusive 2D and 3D sketching. Therefore,
we created VRSketchIn, which provides unconstrained 3D mid-air
sketching as well as constraint 2D surface-based sketching. Before I start
to show you VRSketchIn, I want to discuss about related work and show you
what inspired our system. The first paper I want to present to you is SymbiosisSketch.
It was presented on CHI 2018. Here you can see that it is possible to
immersively draw in mid-air and also have the link to life-sized objects as it
is an augmented reality system. It is furthermore possible to define drawing
surfaces in mid-air. They can be curved or flat.
They are then linked to a tablet, and when you draw on this tablet, you can see
how this is also mapped to the drawing surface in mid-air. This system inspired our
design space, and it can be grouped by it. The next system I want to show you is
TabletInVR it was presented at CHI 2019. It combines mid-air gesture input
with touch input on a tablet. With these interaction metaphors, it's possible to
create and manipulate primitives like this cube here in the video in virtual
reality. This system also inspired our design space, and it can be grouped by it.
Before I want to show you everything in detail, I want to give you a brief
overview about our contributions. First, is the definition of a design space
which is based on expert interviews, 10 in total, as well as the presentation of
interaction metaphors which arise from this combination of 2D and 3D pen- and tablet-based
input in virtual reality. The second contribution is the
implementation of VRSketchIn. This combines the 3D mid-air sketching
interaction metaphors with the 2D surface-based interaction metaphors on
the tablet. And the third contribution are first insights regarding the usage
of VRSketchIn, which are based on a 2-hour long usability walkthrough with
six participants. Now let's show you our design space. We created this design
space to systematically explore the combination of 2D and 3D pen- and tablet-based
input. We conducted a morphological analysis with five
iterations to create the final design space. We see three use cases for this
design space. The first use case is the ideation of new interaction metaphors. The
second use case is to investigate interaction metaphor gaps in existing applications.
And the third is to categorize prior art and identify underexplored parts. We did
all these three steps during our creation of VRSketchIn and
present a populated version of our design space in our paper. On this slide
here, now you see an overview of our design space. This has two dimensions, and
the first dimension is based on the used hardware, which is called input devices.
The second dimension is based on expert interviews and related work. It is
called sketching operations. You get in our paper a detailed overview about the
different interaction metaphors we group inside this design space. Now I want to
show you the system VRSketchIn. It's an artifact contribution according
to Wobbrock et al. 2016. We selected the implemented interaction metaphors based on
our design space, and we made sure that all parts are covered with VRSketchIn.
As Hardware, we used an HTC Vive Pro, a Wacom Intuos Draw tablet,
and used OptiTrack for the tracking, which allowed us to attach very light-weight
trackers to our devices. To create our customized pen, we used a Logitech
presenter to have button input in 3D mid-air, disassembled it and put it into a
custom 3D-printed case and also added the Wacom pen to this custom
3D-printed case to seamlessly switch with this pen between 2D surface-based
sketching as well as 3D mid-air sketching.
The most direct way to sketch in an immersive environment is 3D mid-air
sketching. Just put your pen at a position and
press the button to create a stroke as you can see it here in the video. The
drawback is that you have to manage six degrees of freedom at the same time, and
there are no constraints. That's the reason why we implemented several
interaction metaphors to support sketching. Sketching with reduced degrees
of freedom provides higher precision. That's possible with drawing surfaces as
shown here. Just place them in space by the tablet position and orientation, and it's
also possible to scale and change the distance of them. Everything that is
sliced by the surface is shown as preview on the tablet. Sketching on the
tablet creates primitives in space at the position of the drawing surface. As
example, you can see here how this is done to create the sea around the ship.
Of course, it's possible to switch back to previously created surfaces.
With World In Miniature, it's possible to interact with the whole space in the
vicinity of the user. That's possible because it's attached to the tablet. This
also allows easy perspective-taking, because you can just rotate the tablet.
When you interact with World In Miniature, then it freezes its current
position, which allows a more stable and precise interaction. All interaction
metaphors that are possible in the real space are possible in World In Miniature
as well. Portals and extrusion are further interaction metaphors.
Extrusion is a very common feature because it allows to easily create solid
objects out of strokes. Portals are used like virtual cameras for perspective-taking.
They can be created like drawing surfaces, and the camera view is then
attached to the tablet. To evaluate our system, we contacted a
usability walkthrough. We've recruited six participants through convenience
sampling. Sessions were between 90 to 150 minutes long. You can see the
sessions and what the users sketched in this video. Our usability walkthrough had
three phases. The first was an introduction and training phase, where
everything was explained to our participants. This was followed by a free
sketching phase where they had to sketch a beach house on an island in the sea.
And that was followed by an interview and a questionnaire. We conducted a
thematic analysis and grouped similar answers. In this video, you can see one
example from our participants.
And that's another example. I want to show you our results in a brief overview.
For 3D mid-air sketching, we saw that's a powerful tool,
precision is lower of course, but it's sufficient for sketching most of the
time. Drawing surfaces were used for inorganic parts like straight lines or
walls most of the time. World In Miniature is good for perspective-taking. Interaction is
also possible, and that's useful to interact with distant areas, but
precision is lower. We conclude there are distinct use cases for 3D unconstraint
mid-air drawing as well as 2D surface supported constraint drawing. We observed
that they were used disjoint most of the time. We think that's an interesting
observation, and it should be investigated further if this is due to
the interaction overhead to switch between interaction metaphors or if
there is any more fundamental reason behind this observation. I want to
conclude my talk and show you this beautiful scene of three islands created
by our participants and the ship in the middle, which was created by one author.
Thanks for watching my talk. Do not hesitate to write me questions and of
course enjoy sketching.