Hi Everyone, my name João Diogo Falcão and today I will be sharing the work I’ve done with some collegues of mine:
Carlos Ruiz, Adeola Bannis, Prof Hae Young Noh and Porf Pei Zhang
On ISACS and in-store autonomous checkout system for retail
Well, why research into autonomous retail? The Waiting line
It is such a prevalent aspect of everyday life, in fact, Americans spend roughly 37 billion hours each year waiting in line
Collectively, employed Americans reported burning 4 4Bilion potential work hours per year just waiting in line
Based on the average national hourly wage this makes the cost of wait time experienced by the U S population at $108 billion
Imagine gaining some of that time back to enjoy with your families
On the other hand, convenience stores face out-of-shelf stockout rates of 5%-10%
This means that 4% of sales are lost because the products were not in the shelves for shoppers to pick even though the store had inventory in the back
This leads to $93B annual losses in North American alone
Since this is such a big issue, others have looked into it
Others have automated the retail experience through self-checkout stations
These shift the burden of scanning the products to the shopper, leading to an increase shrinkage, due to intentional and unintentional theft
However these require lower Human Resources on the part of the store
Another approach is to tag every product with a sensor
This approach is very accurate but too labor intensive to be practical
A third technique is to instrument the infrastructure with cameras and sensors
This approach is perceived to be inaccurate but shows promise in removing the human labor
We will focus on these types of techniques
Well, isn’t that what Amazon Go does?
Not quite, notice the store clerks in orange
Amazon Go stores are not unmanned, and several articles point to amazon relying on human-in-the-loop
While impressive Amazon Go stores are developed under closed doors
So we present an open and transparent analysis of ISACS, an in-store autonomous checkout system for retail
By combining modeling of the physical elements of the store, such as the floor, shelves and gondola locations,
with modeling of the people and their interactions with the store
and leveraging the knowledge of where the inventory is located,
ISACS is able to automatically track the movement of products from the store to the shoppers and produce a receipt while the customer is still in the store
Here it is in action
We model the physical elements by measuring the position, rotation and size of all static elements in the store
This gives us a 3D understanding of the store
On top of that we model the products based on the sensed characteristics
In this case we model based on weight
Here we can observe that different products share similar weight distributions and that by weight alone with would not be able to distinguish the products
Here is the system diagram for ISACS
By leveraging the physical store model, weight sensors on the shelves and the cameras in the ceiling We can keep track of the inventory changes
This is done in the “weight change-based item prediction”
by combining the understanding of where each product is located on the shelves encoded in the Product Layout Model with the Product Weight Model
ISCAS further track every person through the cameras
It computes where each person is in every frame and then triangulates them into a 3D world
We now are just missing the interaction between people and products
For that purpose we model the interactions which enables the spatial and temporal multiple human to multiple object matching
This is done in 4 steps
First: Finding when and where an interaction occurred
Second: detecting the closest person
Third: looking at the motion of that person
And Forth: reviewing the past interactions of that person, to see if this predicted interaction makes sense
With this block we now can Predict “who picked what?”
Here is an example of the physical and people models in action
In the corner we can see a 3D representation of the actions taking place
Notice that in certain angles the hands are occluded
However in the 3D model everything is at view
In order to evaluate ISACS we deployed it in a real operating store of 800 sqft with 1653 unique products
We instrumented the stores with 52 cameras and 58 gondolas worth of sensors
Futhermore, we ran and improved the system over the span of 13 months
Having and average of 140 daily transactions ISACS achieved up to 96.4% receipt accuracy
Details on the improvements performed to the system as well as the impact of those in the accuracy are described in paper
We want to thank Carrefour and AiFi for providing the opportunity to research in a real operating store
If you have any questions feel free to send me an email