hi I'm Hui Ye from city university of hong kong
i'm glad to share our paper here ProGesAR: mobile AR prototyping for proxemic and
gestural interactions with real world IoT enhanced spaces.
this work was done with Prof. Hongbo Fu.
real-world IoT enhanced spaces usually involve diverse proxemic and gestural interactions
these interactions are important to increase the life convenience in people's daily life
designing such interactions is emerging needs in IoT industries. to prototype such interactions
designers usually draw 2D storyboards
make videos, using mocap systems and doing programming to specify interactive behaviors
to avoid using complex hardware setup and coding skills some techniques have explored to use the
miniature or the virtual scenes to simulate the real world environments and usually they are
using the familiar skills familiar working schemes of designers
but the lack of connection between the created prototypes and the real scenes make it difficult
to prototype physical interactions happening in the real world environments
recently augmented reality has been used to design AR prototyping tools and authoring tools but
there are not for prototyping full-body proxemic and gestural interactions in real-world spaces
so our goal is to explore a novel tool for prototyping such interactions in the real world space
using mobile AR to make the prototyping happen in situ easily by designers
since there is no existing tools or works about prototyping such interactions we first need to
figure out their existing design challenges so we conducted depth interviews with five design
experts by discussing with them about the prototyping tools and the workflows
their common challenges and problems during prototyping
and their expected features of a novel prototyping tool
so from the interviews we distilled five main issues first is that all of the interviewees
believed that there is a huge gap between the 2d storyboard and 3d real scenes
so the lack of physicality make them difficult to imagine how the 2d scenes will correspond
to the 3d real scenes and second all of the participants think that
they had usually used their multiple perspectives in their prototyping workflow but they find
it difficult to switch between different tools to use multiple viewpoints especially they think they
thought that first-person view and third-person view are both necessary for testing viewing and
examining their created prototype results because the combination of these two viewpoints will
give them a global expression and will allow them to experience from a potential user's view
and then they think they expected
a tool to let them repetitively retest and redesign their design ideas
then they also think they expected a tool to help them create create
interactive prototypes with a low cost finally regarding prototyping device
they thought different mobile devices had their pros and cons they thought
mobile phones and tablets are more suitable for rapid prototyping in their daily lives
based on the results we distill four design considerations for a novel prototyping tool
first the tool should be used for prototyping for real-world things
and real users in situ in mobile AR second it should support first-person view authoring
and both first and third-person view testing third it should support retest and redesign easily
finally it should support to create interactive prototypes at a low cost
based on the four design considerations we design and develop ProGesAR
a mobile AR tool for prototyping proxemic and gestural interactions in real-world spaces
it supports four spatial events and they are represented by two
kinds of event proxies in our system and two testing views are supported in ProGesAR
now we will use an example to illustrate how ProGesAR works
in general designers can follow four steps to prototype their desired results
first she creates the wi-fi asset from the asset set and puts it in the desired
location then assigns an appear effect to it then creates a bulb asset from the set
with a brightening effect
then the designer creates a music asset to the scene
with a play event
then the designer creates a location-independent proxy
and input a threshold for specifying a distance event then the designer creates a location-dependent
proxy and select the gesture button sit button to create a sit event and create an orientation event
after that the designer first selects the square proxy and the appear effect of the wi-fi asset
and maps it then creates a mapping between the location+sit+orientation event
to the play effect of the music asset and the brightening effect of the bulb then move
to the testing mode the designer walks towards the sofa and the wi-fi asset appears
when she sits down on the sofa with facing forward the bulb asset
is brightened and the music is playing and she asks another test subject to enter
the AR view for testing the created prototypes our system is implemented on apple ARKit and
we train svm classifiers to detect the different events please refer to our paper for more details
and here is another result created by ProGesAR
so we proposed ProGesAR, a mobile AR prototyping tool for proxemic and gestural interactions
in real-world IoT enhanced spaces. if you have any questions please feel free to contact us. thank you