Hello everyone, my name is Evgeny Stemasov and I'm here
to present our paper named ShapeFindAR or Shapefinder.
It is meant to be an exploration of in-situ
spatial search in the context of personal
fabrication.
Let me start with a brief look at what we built to do
that. ShapeFindAR is an augmented reality
application written for HoloLens 2, connected to a
database of 3D-printable objects. It supports users
in searching for their physical context by allowing
them to sketch out the geometry they desire, as
seen here, or by extracting labels to use for text
search based on their environment. The entire
process happens at the future location of the
object.
First of all, a bit of motivation and reasoning for
this approach. This is a very simplified depiction of
how we get to physical artifact in the context of
personal fabrication, we design it, fabricate it, and
then we can use it. We could also go ahead and open up
the fabrication step a bit and replace it with
something more general like acquisition. With our work,
we wanted to focus on the first stage, which may be
described as design. Now design may be for instance,
3d-modeling, but we can treat retrieval, remixing,
or parameter input as design activities. Nonetheless,
with ShapeFindAR, we explore the search and retrieval
aspect. We do so in the context of physical
artifacts, ones that we fabricate or buy at some
point.
In the personal fabrication space, there's a commonly
used database or community - which you're probably
familiar with - named Thingiverse. We have
outsourced design effort, which is a lovely entry
point, especially for novices.
Let's take a look at an example. We want to have
this kind of vase with handles, that's a valid
search term. If we have the domain knowledge, we could
also enter amphora as a term. Looking at
Thingiverse, the first query does not yield fitting
results upfront. The second one which requires some
domain knowledge does fare a bit better here. The issues
we saw here are that these type of searches require
domain knowledge for ideal results. They are also
disconnected from their physical context, and they
require a conversion of spatial requirements such as
sizes or proportions to text. This is even then the
case if the user would have a rough understanding of
their desired geometry.
So how can you support users in formulating queries while
they are searching for physical objects? How can
you leverage their physical context and environment for
that, and how we determine fitting results based on
that?
Before we dive in, I'd like to briefly go over some
research areas and works that have inspired us to
develop ShapeFindAR. First of all, we have lovely works
that use different modalities to describe
physical artifacts for instance, gestures, voice,
or sketches. This was also done for search and
retrieval interfaces, which to the end user may look
fairly similar. There are also approaches to situate
search interfaces within users contexts embedding
context information for retrieval. There are also
works that have leveraged XR or spatial computing to
support design and fabrication processes. This
enables in-situ design and interaction which may
reference existing features more easily. Lastly, we have
an array of tools that aim to make design and
fabrication more accessible to end users. We consider
approaches that are based on databases to also fit into
this category.
Alright, let's now talk a bit about the concept and
the system themselves. First, the process we are
aiming for: we have the user, the physical context
they are in, and we want to formulate a query.
ShapeFindAR allows users to enter their known search
terms of course, it also allows them to extract them
from their surroundings based on machine vision.
Users can also sketch out the geometry they may want
or trace objects present in their environment. This
enables them to formulate spatial search queries.
Of course, we can combine these two modalities, which
allows us to be fairly ambiguous with our search
terms for instance. We can also iterate and refine our
queries, preview them in the context and lastly fabricate
the object and permanently add it to our physical
context.
The user facing side of ShapeFindAR was built for
the HoloLens 2 using Unity and the MRTK. We can of
course type in search terms and browse the results, we
can also preview them in situ. We can also extract
search terms from the context based on the frame
the user sees, we crop the image and send it off to the
Google Cloud Vision API. We then receive a set of
suggested labels.
We can also sketch out the geometry we might want in
3d, as you can see here and use that as a query. When we
have received our first results, we can use them to
iteratively formulate new queries, for instance by
adding new sketched features to objects. Now that I've
talked about what the users see and can do: a quick
overview over the backend implementation, we use
MongoDB, flask and trimesh for that. We used
MyMiniFactory and Thingiverse as data sources
to populate our own database. It consists of
approximately 4000 objects, which were then
postprocessed to enable geometry based queries. Each
sketch that a user submits is converted to a voxel
representation, it is then compared to fitting
candidates found in the database. In the backend, we
converted all models in our database to voxelized
representations. We do the same for the user's sketch,
and they are aligned via ICP, similar to the Data
Miming paper, and we calculate a score based on
overlap. To accelerate queries, we are also pre
filtering the objects based on the proportions of their
bounding boxes. This is a method we settled on, but
there are many viable alternatives.
Let's take a look at some walkthroughs we did with
ShapeFindAR.
We might have an Arduino clone lying around. The name
would be an ideal query, but it is not written anywhere.
Therefore, this demands domain knowledge. So we
frame with the object and we receive suggested labels to
use as text queries, we can then preview results
in-situ. This depicts the transfer between physical
context and a textual query. We can also iterate on
queries, refining them with the results we get along the
way. Say we have found this bowl but want to have
something more elongated. We sketch onto this result and
submit this as a new query. We may then receive results
that have comparable proportions. This addresses
the iteration part of spatial queries.
We may also trace relevant features to formulate
spatial queries. For instance, this flowerpot. We
can then submit the sketch as a query along with an
abstract term, and of course preview fitting results.
This depicts the transfer between physical context and
a spatial query.
Our work certainly has its limitations and has avenues
for further improvement. The processing time of our
server is not ideal, and the matching algorithm does not
always perform as desired. Our dataset is fairly
diverse, but also smal as it is a subset of databases
like think Thingiverse.
For future work, we plan to do user studies to
understand how search approaches may change with
such a tool. And we'd like to add more modalities to
the search such as 3d scans.
To wrap it up, we presented ShapeFindAR, an AR tool,
enabling in-situ spatial search. Searching and
finding is a valid entry point to personal design and
fabrication. And it can be made more powerful: by
lowering effort and easy access to model
repositories. In the context of fabricating and acquiring
physical artifacts, their future context becomes an
indispensable feature to use and rely on.
And the vision behind this is to let users express
themselves in the modality they are comfortable or
proficient with and let them mix them how they want or
depending on the task they face.
That's it. Thank you very much for listening. Thank
you to all my wonderful collaborators, too.
For more details, please check out the paper