- Thanks for the introduction.
Today I'm gonna talk about AirCode Unobtrustive
Physical Tags for Digital Fabrication.
This is a collaboration with Avinash Nair,
Shree Nayar, and Changxi Zheng.
Tags are everywhere, just to list a few
commonly seen examples here, from luggage tags
boarding passes, product barcodes, to mailing labels,
tags are essential part of our life.
They make our travel and everyday life much easier
because they carry metadata to identify
objects to create hyperlinks between
physical object and digital information.
For example, linking a barcode on groceries,
to digital pricing and inventory stock information.
And sometimes even guide interaction,
for instance showing which side of the package
is up, or whether it's fragile.
However, with all these great features
there's one caveat, tags alter appearance
and compromise on aesthetics.
For example, on this beautifully designed
movie poster, although the QR code can conveniently
redirect us to the movie website, the intrusive
QR code greatly compromise the aesthetics.
Similar to 2D art designs, putting traditional
tags on 3D object also change the appearance
and introduce undesirable visual intrusion.
In this talk I'll introduce an unobtrusive
tagging method, AirCode, which are invisible to naked eyes,
yet machine readable with computational imaging methods.
They're a long line of research on tagging methods,
I'll discuss three most related ones.
The first one is InkaStruct.
Willis et al embed tags using multi-material layers
and decodes them with Terahertz imaging.
Since different material responds differently
to Terahertz signals, tags can be decoded
using specialized imaging and sensing equipments.
The second work is Acoustic Barcodes,
from Harrison et al, they introduce
structural patterns of physical notches on objects.
When swiped through the bar code formed
by the notches generates a sound wave
which encodes the tagging information.
To facilitate interactions instead of using tags
Affordance++ guides the user arms using
electric muscle stimulation.
The design interaction in Affordance can
be conveyed with real time guidance.
And different from previous work we will
like to embed tags inside the object
without changing the visual appearance.
To achieve invisible tags, we'll first look at
how light transport and visual appearance works.
Suppose we have a solid plastic slab,
a light source, and a camera.
For all the rays that reflect directly on the surface
we call them direct components.
This direct component dominates the visual appearance.
And when lights penetrate inside and scatter
it's called subsurface scattering.
It happens in all non-opaque material
like plastic, fabrics, papers, or even our skins.
For the rays that arrive through this subsurface scattering
we'll call them global components.
And in contrast to the dominant direct counterparts,
the global one is usually very weak in intensity
because energy is lost during the scattering process.
When we take a single photo of these two parts
and combine together and the direct part
contributes to the majority of the appearance,
a natural idea is to embed unobtrusive tags
in the global component while keeping
the direct the same, hoping that overall
change is not perceptible.
Immediately, two questions arise.
How can we guarantee the tags are invisible?
We can imagine that if the global part
is changed drastically, the tags will still
become visible, and second, since in photos
the direct and global are always combined together
how can we separate out only the global parts?
Luckily for the second problem there is already an existing
separation method that solves it.
By projecting different structural
live patterns, for example in this case
we used checkerboards and we take series images.
We can computationally separate out
the direct and global components.
And now we use this photo of a leaf
to show a typical separation result.
On the left is a direct component
on the right is global.
We can see the direct contribute to most
of the visual features, whereas
the global image capture the material difference
in leaf veins through light transport and scattering.
This is the kind of effects we are looking for.
The global components captured on the
laying material difference.
And now let's look at how we can actually
manipulate the global components during fabrication process.
Our key idea is to place air pockets
inside the object, since the air pockets
are beneath the surface, the dominance
direct component is unaffected.
On the other hand, in a global component,
light scattered beneath the surface.
The global components are different at a
different regions will be different.
Our project aims at finding the best
location to put this air pockets.
As in the illustration here we plot the
separation we got for this 1D example.
X axis is the position, Y axis is the intensity.
Since air pocket do not change the direct components
the direct intensity stay constant over this x range.
As for global component we'll notice a drop
over the air pocket because
the change of light scattering behavior.
And making use of this intensity difference
we can encode information.
If we divide object into three regions
based on the global component intensity
we can encode zero or one in each region.
And from a top view, this one, zero, one pattern
look like this, and white square
is the one and dark square is zero.
Using this basic idea we can construct
the full AirCode Layout.
And this layout is largely inspired by QR codes
it consists of three main regions.
Markers, data Bits and calibration bits.
The four markers are used to locate a tag
we use circular shape instead of squares
to increase the decoding robustness
and the blur and noise.
The calibration bits are used to reduce
distortion and improve accuracy.
The remaining binary cells are data bits
to store user specified information.
Putting this AirCode into a printed piece,
and this is our capturing system,
notice the AirCode has several distinctive benefits.
First is embedded during fabrication process.
Which means no post processing
or extra electronic installation is needed.
Second, the tags are invisible
and unobtrusive to the input geometry.
Third it can be decoded easily using
consumer level camera and projector set up.
And now I will talk to you about how to make sure
the embedded marker and bits stay invisible.
This invisibility problem is closely related
to concept of contrast threshold.
The definition of contrast threshold
is very straightforward, it's the minimal contrast
required to see a target reliably.
In other words we want the change
in our global component to be smaller
than the threshold, this way the embedded tags
can't be seen reliably.
And it turn out this invisibility problem has been
widely studied in physiology and vision research.
Two main factors contribute whether
or not we can see a target with our naked eyes.
Contrast and intensity patterns,
let's look at each of the factor.
Contrast is defined as a difference between
maximum and minimum luminesce
normalized by the space average.
This is very easy to compute given an
image and their pixel values.
As for the intensity patterns, various patterns
have been studied such as sin wave, square wave.
And notice that this primitive shape
closely resemble the typical global component
of a fabricated objects.
So we adopt the results and the
statistics from the vision research.
And in this paper the contrast threshold
vary from 0.1% to 10% depending
on the pattern features and overall luminesce.
To interpret and use this result,
we need to be able to predict the contrast
by simulating global component of air pockets.
And in order to do that we use layered
scattering model commonly used in rendering
proposed by Donner and Jensen.
Let's look at the first single layer case.
For each layer there are two profiles,
Reflection profile R and Transmission profile T.
Both profiles describe the ratio of
outgoing energy between incident
and outgoing race, and D is the distance between
incident and outgoing directions.
If both the incident and outgoing race
on the same surface is a reflection,
otherwise it's a transmission.
This profile can be easily measured and
stored in the database.
To incorporate air pocket we need to look
at how multi-layer profile works.
And here are two layers with different reflection
and transmission profile, R1T1, and R2D2.
The goal is to compute the combined
reflection profile R because that's what
captured on camera and that's what we perceive.
If a ray comes in and leaves at the top surface
without transmitting to the second layer
the contribution is simply R1, this is from
the definition of reflection profile.
In a more complicated case when
the rays scatter into the second layer
and reflect back the first layer
and comes out the contribution is T1 times R2 times T1.
The first transmission is T1, and the reflection
from the second layer is R2,
and the final transmission passing through
the first layer again is T1.
And this is the one bounce scenario.
Similarly this extends to two bounces cases
and so on and so forth.
This expression can be simplified
and computed efficiently with analytical formula.
And this also generalize easily to multi-layer
profile calculations with air pockets
and printing materials.
Putting the pipeline together given
any multi-layer configuration, we can
compute the contrast efficiently
and check if it's within the contrast threshold.
In our paper we discuss more on the parameter
optimization, error analysis and the layout design.
If you're interested I encourage you to read our paper.
Now I will talk about three applications
enabled by AirCode, metadata embedding,
robotic grasping, and paper watermarking.
The first application is metadata embedding.
Inside a square piece we encode a tags message
which is invisible on the normal lighting,
yet can be revealed with separation methods.
And here's the same piece, holding in my hand.
In cutting it open this visualized internal
air pockets, note that the AirCode can be fabricated
with a single material 3D printers.
And here are the directs and global components.
We can see the global components normalize
just for visualization purposes,
and we encode word visibility.
AirCode can be further used to encode other metadata
similar to those stored in photos.
Now the three models can encode time,
location or what kind of printers
that they are fabricated on
without changing the appearance.
And taking this one step further,
AirCode tags can also be used to create
hyperlinks between physical designs
and digital information.
In a global component image of this statue piece,
AirCode embeds a hyperlink to the
corresponding Wikipedia page and we invision
that this connection between physical
and digital world can be further explored
to we can embed, for example, copyright information,
authenticity verification, or even convey
accordance for functional object.
AirCode tags can also help robotic manipulators
to interact with manmade objects.
In robotic grasping task, a challenge
is to recognize an object, estimate it's pose
and decide where to grasp.
For this unconventional triangular drawer,
by embedding AirCode on each of the side faces,
AirCode can not only recognize the object
but estimate the pose and decoding
and doing the recoding and rectifying process.
Different from traditional grasping pipeline,
AirCode solved all of these challenging
issues by embedding information in the same AirCode tag.
And this is a robotic grasping example.
Is it playing?
Oops.
sorry.
Now the AirCode is decode and recognizes
the triangle drawer knows that it's the three model
and it's supposed to grasp on the red edge,
and it also knows the pose of this drawer.
And since we do not use traditional
object recognition or dab stancing
even if the handle is fully occluded
in this case, we can still recognize the drawer
and predict the grasping location and succeed a grasp.
We also test AirCode on curved surface on a 3D printed mug.
Again, AirCode can robust, identify the handle,
even when it's fully occluded.
Know that AirCode tags are generated
in the process of fabrication so the robot
can identify and manipulate the object
immediately after it's fabrication.
And since AirCode is based on subsurface scattering,
this means that the theory can naturally extend beyond
3D printing materials, for instance
paper boxes, mass produced plastic containers,
or even cloth and fabrics.
In general, as long as the materials are not
fully opaque it's possible to explore
subsurface light transport for tagging.
As a demonstration here we embed watermarks
in a stack of papers.
The embedded text is invisible.
And reviewed in the global component.
And this is how we construct it.
The bottom two layers are simply blank papers,
follow by AirCode layer which is carved
by a stencil to hold or heed a message.
And the top layer is printed with
regular, visible text.
And here are the separation results.
To further clean up the image,
we can create a binary mask from the direct image
and apply standard image impending air with them.
To reveal that the underlying text clearly.
While this demonstration is a simple experiment
we're excited about the potential to
generalize to a wide range of materials.
In summary, we propose AirCodes to tag physical objects.
This unobtrusive tags are embedded
during fabrication and can be decoded
robustly with a camera projection set up.
And our theory generalize to all non-opaque materials.
For future work, currently, the separation
takes several minutes due to the
structural light pattern sweeping.
And we believe this will improve
as we have more powerful projectors
and better camera senses in the future.
Another example that we show are smooth surfaces.
For shapes like this with very bumpy surface
or very thin structure, it will be challenging
to directly adopt AirCode, and more complex
geometry is a promising future direction.
With physical tags, another issue is
long term preservation, the property of plastic
material changes over time,
especially for the 3D printed ones.
For example these two were exposed to different humidities.
In our experiment the models have been printed
for almost a year and the optical property still holds
for robust decoding but we need rigorous tests
to understand the long-term impact.
Last, since AirCode is embedded during the fabrication
process it's almost impossible to change
once it's fabricated, so we like to investigate
ways to change it without reprinting the whole structure.
I'd like to thank our colleagues for their supports,
and to Thingiverse users for sharing their models,
also want to thank NSF and Adobe
for founding this research.
I have some of the prints with me here,
please drop by at the end of the session
to challenge yourself to see the invisible.
And with that, I want to thank you for your attention.
(applause)
- [Host Speaker] Alright, if you have a
question please come to the microphone.
- [Audience Member] So you briefly touched on
this in the summary of the more complex
3D structure because all you showed was like the 2D,
tags basically.
What's the problem, is it more from the vision
side or do the pockets need to have a certain size,
or what's the challenge of the structures?
- Currently it's most of the printer resolution,
yeah, that's limiting the AirCode bit size, yes.
- [Audience Member] Okay.
- Yeah.
- [Audience Member] Hi, I noticed that in the
photos that you showed, a lot of the objects
are just white and color, have you had
experiences trying this method on different colored objects?
- We haven't tried other color of materials
mainly because the printer that we have
only prints white material, but the theory
generalizes to other, like the paper we show
is actually in the yellow shade,
and it works, too, so I guess it's
generalized to other colors, too.
- [Audience Member] Okay, thank you.
- [Audience Member] Hi, Johnoton Debrid, Tactual Labs.
It's great work.
I was just wondering, so some of the examples
you talked about in the future of some mass
produced items or paper that were injection molded
or produced in a mill, those obviously aren't conducive
to the customization of each individual object
that a 3D printed object would be.
So I was just wondering if you had thoughts of
about how this could be integrated with
something that wasn't actual mass production,
like injection molding versus putting on a label
or laser etching a label, or something that's
done after the manufacture of the mass produced objects.
- Yeah we haven't really thought about the ink
like modeling process, but in general
as long as the plastic containers,
it doesn't have to be single material,
it could be multi-plastic that has different
optical properties, it's just that in our case
we can enable this in a lot of 3D printers
that only support single materials.
So, as long as the mass produced plastic can support
multi-material combination then
yeah, our method should work.
- [Audience Member] Hi. Thanks for the presentation,
it's really amazing work I like it a lot.
So this is more like a thought experiment,
and I guess also it's more of a
subsurface scattering question, did you try anything
with different materials
like I don't know,
like embedding water instead of only air,
or like squishy materials, or anything like that?
- Yes, actually we tried with the support material
that's usually come with most 3D printed piece,
and the problem with that is, in the global image
it's usually blurry because the reflection of index,
they're very similar between support and model material.
And that's why we chose air because that has the most
distinctive difference. Yeah.
- [Audience Member] Thank you.