- Amazon, as you know, has done a lot of work
in the online world,
and we are setting the standard
for what constitutes convenience.
Our challenge, when we started
the physical retail organization
was what is our unique spin on convenience
in a physical environment?
(upbeat music)
My name is Dilip Kumar.
I'm a Vice President in our physical retail organization,
and I've been at Amazon for the past 18 years.
I was Jeff Bezos's technical advisor for a couple years,
and when this notion of physical stores started
way back when, in 2012,
I was one of three people
who was chartered with figuring out
how do you make physical stores different
and convenient for customers?
When we launched physical retail,
our first product that we launched was Just Walk Out.
That led to the Amazon Dash Cart,
which, again, was another way for us to provide convenience
for customers to skip the checkout line.
But while that was all focused
towards skipping the checkout line,
there was another parallel initiative that was brewing,
because there were many retailers
which had point of sales
and customers were still waiting in line.
So our primary impetus for developing Amazon One
was to find ways to speed up the process
of customers having to not have to wait in line,
even if there was a point of sale.
Amazon One is a fast, convenient, contactless way
for customers to identify, enter, and pay
in any physical environment.
So once we wanted to create Amazon One,
there was a range of biometrics that we could choose from,
face, voice, fingerprints.
We selected palm, because it had several characteristics
that were very suitable for us.
The first one was, we wanted it to be contactless.
Now, several biometrics met that criteria.
The second criteria that was very important for us
is that we wanted the biometric to be private.
When you look at a palm,
you can't ascertain a person's physical identity
by just looking at a picture of their palm.
Unlike face, or even your voice,
which can give you clues about the person's identity,
a picture of your palm doesn't give you a clue
as to who the person is.
Third one was, we wanted this
to be a very intentional gesture.
What makes a very simple, intuitive gesture for customers?
That's where we started.
People are used to holding a phone over a scanner
in order to be able to identify themselves,
so we said, well, let's remove the phone.
People are used to this gesture
of holding something over a scanner,
so we can just eliminate the phone from that
and just have people hover their palm over it.
It wasn't a passive biometric,
like your mere walking into the room or standing in line
doesn't trigger a biometric to be identified.
So we wanted the customer to always be in control
over where and when they choose
to either identify or to pay.
And the last one, which is very important for any biometric
is if the biometric isn't accurate,
if it doesn't actually do the job
of being able to identify the person,
then it's not a very suitable biometric,
and we felt that the palm and the vein
were very suitable biometrics,
but that also required us to do a lot of research
to get to that stage to make it a suitable biometric.
Then the question was
that how do we assemble the right set of skills
necessary for it?
The first one was the hardware, or the device itself.
Now, in order to be able
to capture the right kind of details from the palm,
as well as the subsurface images of the vein,
it involved very sophisticated camera and optics
that we needed to build
in order to be able
to get to the right level of sophistication.
From a software perspective,
we wanted this to be extremely intuitive
and very easy for customers to understand.
All a customer needs to do
is come in and hover their palm over the device.
But we needed our software to be very quick,
we needed an ability to provide customers guidance
on where they needed to hold their palm over the device,
and we needed to be able
to figure out the identification task
of knowing who the customer is
in a very, very small period of time.
That provided a huge amount of challenges,
not just for the software side, but also the research side,
because the research was all about figuring out
how do you take two different biometrics,
the palm, the surface characteristics
that you can actually see on your hand,
as well as the subsurface characteristics
that are just below the surface, the vein structure,
and how do you use a combination of those two
to triangulate, to figure out,
and have very high accuracy for this biometric.
Unlike face or voice, which has plenty of data available
in order to train computer vision algorithms,
there wasn't any publicly available data sets for palm,
so we had to create our own.
Now, one of the challenges
that computer vision algorithms in general face,
and it's not specific to biometrics,
is unequal representation of data
can cause biases in the algorithms.
So for us, we had to create synthetic data
to create subtle variations in your palm and vein structure,
and to be able to train our algorithms
with millions and millions of data points
without requiring millions and millions
of physical customers using the device.
In order to do that, we created these artifacts in our lab
which allowed us to figure out very quickly
what constitutes a sharp image,
a sharp image referring to,
are we getting all the detail necessary
for us to be able to separate one palm and vein
from another palm and vein,
and how does that vary,
based on how far a person holds their palm over it,
whether it's cold, or whether it's hot,
whether it's belonging to different times of the day.
There's a variety of different things
that cause your palm and vein structure to change over time.
We wanted to be able to capture those subtle differences,
and in order to do that,
we built these motorized testers
that use a spatial frequency response as a metric
to determine whether is this a good image quality
that's necessary for us
in order to be able to identify this palm
if you are hovering it over for just a fraction of a second.
The problem with palm and vein data is
that you want your synthetic data to be realistic enough
so that your algorithms don't get confused
by things that are never seen in anybody's palm
or never seen in any vein structure,
because adding more of that just adds noise,
it doesn't help,
so it needs to be realistic,
but it needs to be separate enough,
and it needs to be driven
off of the specific characteristics of the palm and vein
for the different subsegments
in order for us to be able to boost enough data
for each of these separate areas
so that we can get our overall accuracy up.
We look at things where we are unable to identify the person
or we misidentify the person,
and our misidentification rate so far has been zero.
We'd rather be in the position
of not being able to identify
in case we don't have adequate details
rather than misidentify.
So we skew and bias our algorithms
to say that it's okay to not identify,
and ask for other forms of identification,
rather than make a misidentification.
Well, when you're building
an identity and payment biometric service,
security was even just as important, if not job number one.
We have tamper detection and intrusion detection
on the device itself.
If the device gets misplaced, it gets lost, it gets stolen,
there's nothing stored locally on it.
Everything is encrypted.
We don't accept raw biometrics.
Only biometrics that have been generated by the device
and are attested by the device, and can talk
to mutually authenticated endpoints in the cloud
are the only ones that are allowed.
So while we have an end-to-end encryption,
we also have the ability where we shut down.
Only the set of things that are necessary
to talk to the device are the only ones
that are even allowed to talk to the device.
For any payment device,
we had to go through third-party certifications
that indicate that the level of security,
and the confidence, and the assurance
for both, not just for us, but for also customers,
that it's adhering to the level that customers expect
from any identity and payment service.
One of the other things that we did
to boost the security of the device
is something known as liveness detection.
We needed to know if someone is holding a live hand over it
or if it's an artificial or a silicon hand,
and so we created multiple different artifacts
of silicon hands
to be able to figure out what is the difference
between a fake hand, as well as something
where you have a live hand
where you can actually see
the blood flow and the vein structure.
There's a variety of things that you can do
to identify a fake hand from a silicon hand,
and so we built testers to not just identify that,
but we'd also be able to identify things
like people can...
Sometimes people come in with dirt on their hands,
or there's other tattoos, there's other kinds of marks.
So you can actually determine a variety of different things
in order to figure out how to boost the overall accuracy
and the security of the system.
So security is something that we built into this device,
built into our software,
built into all our processes from day one.
Customers have loved the experience,
they've loved how simple and easy it is to use,
and it saves them time when they're waiting in line.
There's many different ways in a physical environment
where one has to stop, one has to identify themselves,
one has to pay, either for loyalty or for paying your bills,
whatever those cases might be.
Amazon One is uniquely positioned
to be the way that customers identify, enter, or pay
in a variety of different environments,
which is secure, it is fast, it's convenient,
and it's effortless for customers.
We're building complex products
where our challenge isn't just
the complexity of building it,
it's to make this complexity recede from customers
so that when they come in and hover their palm over it,
that's the only thing that they have to do,
and then we take care of all the complexity
behind the scenes.
And I tend to focus a lot
on the simplification aspects of it,
because simplification helps us create
very intuitive and very easy-to-understand experiences
for customers.
Humans don't like to subtract.
Humans love addition, they like to add features,
it feels like that they're adding value,
but sometimes, the simplicity of taking things away
is just as powerful,
and while it increases the challenge for us,
it resulted in a great experience
and a very simple experience for customers
when they came in to pay.