hello everyone my name is xun chen from
purdue university today i'm going to
present our work scalar ordering
semantically adaptive augmented reality
experiences in virtual reality
recently semantic understanding
techniques have been adopted in ar
system developments such as synthesizing
spatial movements of ar pads optimizing
ar interface layouts around the user and
generating realistic avatar behaviors
based on the affordance of the engaging
objects
all of this work achieves semantic level
association where the error content
behaviors are tightly associated with
not only the location but also the
affordance functions and semantics of
the physical objects
so the digital augmentation can be more
seamlessly and intuitively delivered to
the end users
meanwhile this works implement
computational algorithm to ensure such
association can be adaptively maintained
in different environments as long as the
same objects are detected this is what
we call a semantically adaptive ar
experience
however the ar content behaviors in
these works are predefined by the
programmers sitting in front of a
computer but in real environments there
are dissimilarities of the room layouts
duplication or absence of the physical
objects which may introduce implicit
semantic variations that are difficult
to be pre-considered during the
programming stage especially when the
designer is not situated in a target
environment
and for the users who have limited
coding experience it's also hard to
design and customize the ar experience
with their domain expertise
we aim to enable designers to create
such ar experiences while fully
leveraging the semantics in all the
potential targeting environments
virtual reality is a good solution on
one hand vr can immerse users into a 3d
virtual scene to intuitively alter the
virtual content behaviors using the
surroundings as spatial and semantic
references it also has this spatial
flexibility so that the user can easily
travel across different scenes without
leaving the current location
so here comes our main idea
using vr we provide ar designers with
multiple realistic virtual replicas of
the physical environments and let them
immersively alter the semantic
association
then we distill the demonstrations in
all the provided virtual scenes as a
mathematic model to deploy the ar
experience in different environments
which becomes our proposed system
scalar now let me briefly talk about
overall idea of the scalar workflow
using scalar the ar designer first
authors the semantic association in two
levels the designer has to define which
physical objects hold the most relevant
affordance with air content we call it
identity association meanwhile the
designer should also ensure the
placement of the ar content is accurate
we call it spatial association for
instance this ar lamp should be placed
at the corner of the coffee table next
to the sofa to deliver its digital
augmentation to this consumer
remember we also want the adaptation
capability
which means the ar content should be
able to respond to the variations in
different environments
we consider two types of variations that
commonly happen around us first the
locations and sizes of the physical
objects can be different we call it
spatial variation
of course this designer need to modify
the air content behaviors accordingly
for instance if all the tables are too
far from the sofa the designer wants to
place the ar lamp on the arm rest of the
sofa
second the quantity of each object can
also be different word objects is even
not there
scalar provides such scenario and let
the designer to modify the design as
well
for example what if there's no table in
a small living room
after the authoring session we record
all the designers demonstration of the
semantic associations and their other
provided scene variations and group them
accordingly for the fitting of the
mathematic model
we call this model semantic adaptation
model which is used to procedurally
determine the air content placement
given a new physical scene i will cover
it later
above is the overview of the scalar
workflow next i will explain some key
implementations of scalar
first how does scalar generate those
virtual scenes with spatial and quantity
variations
after raising a basic idea of the target
deploying environment the designer
invites potential ar consumers to
collect their local environments as
samples using the scene collection
modules of scalar
here we adopt the combination of 2d
object tracking and 3dc understanding to
get both the identity and spatial
properties of the physical objects
ar consumers just follow the guidance to
scan the environments and final outcome
will be displayed as 3d labeled bounding
boxes
then for each collected scene we apply a
genetic algorithm to synthesize more
virtual scenes with the scene variations
mentioned before and we render them
using pre-assigned cad models to
construct the vr authoring environment
now we show the vr authoring interface
of scalar we use an arvd display in a
cooking tutorial as an example our
system allows the designer to explicitly
define identity association using line
connections such as this counter top to
be the primary binding physical objects
of the ar display
in order to eliminate the ambiguity of
the duplicated objects and consider the
multi-binding object cases the designer
can also define reference binding
objects as the additional reference
here the designer wants the ar display
to be always placed on the countertop
near the ranges to play some cooking
tutorial
meanwhile the designer can intuitively
manipulate air contents to define the
spatial associations using surroundings
as spatial reference then the designer
uses the scene cursor to travel across
all the provided virtual scenes and
verify the design initially we apply the
altered association in all the scenes so
only when the scene variation causes
semantic ambiguity or spatial inaccuracy
would the designer enter that scene and
modify the association for the other
scenes the association will be
automatically recorded after the
designer validates all the scenes the vr
ordering session is complete
then our system starts to feed the
semantic adaptation model since the
designer may change the identity
association due to the spatial
variations we first fit an svm model for
each semantic group to predict which
physical object should be used as the
identity association enter the current
physical layout
next for each case of the activated
identity association we develop an
algorithm to determine the ar content
spatial location please refer to the
paper for more details of the fitting
process finally back in ar scalar
deploys the ar experience to a new
environment using the semantic
adaptation model first we check whether
there exists an svm model that takes
some of the involving physical objects
and inputs
here is an important consideration if
our system doesn't find any applicable
svm model we won't render the air
content for instance if the designer
creates a living room decoration ar app
most of the ar contents won't be
suitable for a kitchen environment
in contrast if the system finds a proper
model it adopts the model to get the
activated identity association then
calculates the local spatial attributes
of the air content and renders it in the
environment
for example for the deployment of the
cooking tutorial air consumers scan
their local kitchen layouts using the
same sync collection module and scalar
adaptively render the altered ar
contents using the semantic adaptation
model
thank you for watching for more details
such as the use cases and user studies
please refer to our paper and video