hi everyone I'm Karthik a PhD student at
the University of Toronto
today I'm presenting work and during my
internship with Autodesk research titled
Tesseract acquiring spatial design
recordings by manipulating worlds in
miniature
designers often review their work or the
work of others during the design process
previously researchers have proposed
interfaces to help designers review 2D
design recordings for various purposes
including learning new workflows and
understanding design rationale
recent VR collaborative design tools
have made it possible to create and use
spatial design recordings containing
multimodal data about objects
people
and their interactions
objects can include 3D models textures
animations and entire scenes
as designers compose a new design they
manipulate objects and their properties
creating versions of the design at each
moment which can be recorded and later
retrieved through query
spatial design recordings also record
people as they engage in the design
process by capturing their activities
activities are recorded across frames
including how people move around the
virtual environment what they're looking
at and verbal dialogue between that themselves
are others
spatial design recordings are typically
created in feature-rich applications
that include tools for manipulating
objects such as when a 3D model is moved
so these interactions can also be logged
and retrieved
despite their potential to serve as
design aids searching these recordings
presents many challenges first spatial
design recordings contain multimodal
data about objects people and
interactions
second data occur over multiple frames
and are recorded over multiple sessions
for instance when a designer is making
an interior layout for a client the
process to finalize the design takes
multiple sessions where the design is
reviewed and iterated upon
existing search techniques designed for
searching 3D models and static 3D scenes
do not support searching across multiple
frames
on the other hand existing timeline
navigation techniques for spatial
recordings can support scrubbing or
jumping across multiple frames but do
not support searching multimodal data
further users are restricted to browsing
a single recording at a time
hence the challenge of querying spatial
design recordings lies in their lack of
support for users to express their
search intent to retrieve moments that
consist of multimodal data across
multiple frames in multiple sessions of
recordings
we present Tesseract a worlds and
miniature based system to support
searching spatial design requirements
for interesting moments of past design
activities
Tesseract is built on top of the search
Cube interface which lets users Express
their search intact
similar to a search bar in a
conventional search engine the search
Cube interface supports users in
defining the 3D content that make up
relevant moments that they wish to find
in past recordings
how to define the 3D content depends on
the type of data one wishes to
incorporate in their search
in this work we explore spatial design
recordings from the lens of interior
design
hence to search spatial design
recordings we created four querying
tools
object search
proximity search
Viewpoint search and voice search
object search allows users to add and
manipulate objects into the search Cube
to define relative layouts that might
exist in previous recordings
each object becomes part of the query
through its presence in the search queue
by selecting an object users can include
them in the search prioritize them
or exclude them
when multiple objects are inside the
search queue Tesseract automatically
detects object object relationships such
as when two objects are next to each
other
proximity search lets users define the
location of recorded people and
proximity to specific objects they may
be interacting with or visually
examining such as when designing for
accessibility in support spaces
a resizable selection disk represents
the area around a recorded person in
proximity to nearby objects
users can manipulate the disk to define
the reported person's location
they can also resize the disks to
include more or fewer objects next to
the person
proximity search can also be used to
specify an absolute location within a
room that the recorded person was in
to access this the user can replace the
generic Search cube with a specific 3D
layout that they might have copied from
their current environment
Viewpoint search supports querying by
letting users define their own recorded
viewpoints or areas that a recorded
person may have been looking at either
while stationary or moving around
a recorded Viewpoint acts as a proxy for
a recorded person and the viewpoints
position and orientation can be
manipulated to filter which objects are
visible
to define a sequence of viewpoints that
a recorded person demonstrated frames in
time can be used
when multiple persons or reported people
are present users can add more recorded
viewpoints representing them
each recorded Viewpoint has their own
trajectory defined using individual
frames
lastly voice search enables users to
find events log where people interact
with the design software and objects in
the virtual environment such as when
adding or deleting 3D models
it also supports finding conversations
or think aloud dialogue by designing
we evaluated Tesseract in a 10
participant user study
we provided Target clips for
participants to find to ensure that they
have the same search goal so that we
could observe the variations in how they
use the querying tools
seven target clips were chosen based on
scenarios involving understanding design
rationale reflecting on designs
and learning new workflows
participants then used the search Cube
interface and the associated query tools
to look for the target clip
lastly they used the results of their
search to find the target clip
in the study we collected data about
participants usage of Tesseract and the
associative querying techniques
we also collected qualitative feedback
such as reflections on the querying
tools and the search Cube interface
through questionnaires and an interview
for each task we aggregated participants
resulting search cubes to better
understand their search patterns
for instance in the object search task
the left two we counted how many objects
and the types of objects participants
included in their query
this helped us observe interesting usage
patterns participants exhibited
first we noted that most participants
included objects that were the focus of
the recorded designers from the target
clip
in the case of the first object search
test this included the refrigerator that
the designer was adding during the
clip
more interestingly participants also
included peripheral objects not
mentioned or interacted with for
instance the island counter was never
interacted with in the clip but
participants still included it every
single one of them did this suggests
that these objects is serve as spatial
anchors that helped describe the moment
to be found
if in Focus objects represent the
activity then peripheral objects
represent the context
we also found that most participants
were coarse when trying to replicate the
layouts on the target clips only one
participant replicated the exact layout
in the Target group most opted to
include only partial layouts with some
of the objects
however most part of participants were
precise in their placement of objects
with respect to each other including
their relationships this trend was
visible also when placing the selection
disk in the proximity search task
and recorded viewpoints in the Viewpoint
search test
lastly we observed that most
participants refined their queries after
seeing the search results which is akin
to how people use conventional search
engines today where text queries are
rephrased when the desired result is not
found
hence we found that our proposed
querying tools offer different ways for
users to express their search intent and
to iteratively refine them
participants also found the search cube
interface and querying tools easy to
learn use and efficient together
Tesseract provides a toolbox for users
to express their search intent when
searching for past design activities
to conclude through Tesseract we present
the idea of stage to query
using a miniature based VR interface
across space and time
we introduced four new querying tools
and we showed that using these querying
tools participants were able to express
their search intent and retrieve past
design activities from spatial design
recordings thank you