- At Starbucks our mission is to inspire
and nurture the human spirit.
One person, one cup, one neighborhood at a time.
You may be familiar with some of our 30,000 stores
worldwide, that may be a part of your experience.
We have over 30,000 employees who we call as Partners
who ensure our standing as one
of the world's most admired companies.
In order to adhere to that quality of service and product,
the key focus areas that we focused on are,
making sure we have elevated Partner
and customer connections, breakthrough
in beverage innovation, and also accelerating
our customer engagement across channels, including digital.
However, this story is driven by our ability
to harness data to power these customer experiences.
Amazing teams across Starbucks continue working
on the next breakthrough innovation
to ensure we build the right customer experience
and an enduring brand appeal.
These extend across our multiple domains.
Partners, products, store portfolios and digital.
There are tons of awesome teams working
on big ideas and enabling them to be brewed at scale.
As much as it's about the product,
it's about elevating customer connections
and the experience that is driven
by the convergence of data.
For example, on mobile, we use rapid AB testing
to improve recommendations using reinforcement learning,
models are the BrewKit platform.
On drive thru, we cross sale modifications
using engines like Debrew.
On store applications we serve near real time
transaction data with billions of data points
to internal reporting applications
so that actual store personnel
can monitor transactions near real time.
More importantly, data is extremely crucial
as we extend channels for delivery
and new norms into this new era.
The ability to serve these used cases is
only possible through a sound data strategy.
Our data strategy and guiding principles are built
on three pillars, single version of the truth.
Providing trusted store of information
across customer part of the stores.
Data analytics enablement, as well as, trusted data
to ensure data quality, privacy, security
and the right data definitions to accesses data.
How to execute our strategy we had to deal
with several modernization challenges.
On the architecture side, we were dealing petabyte scale
of data and fragmentation across systems.
Or the ingestion and processing the ability to process
near real time data products and a huge variety of sources.
The inability to implement updates, merges
on fast changing data in an optimal fashion.
Also non-optimal engineering experiences
with services that did not perform at scale.
For example, taking a long time to provision or scale.
Also, on the consumption side,
there's no single source of truth.
A lack of a unified user experience
and an impedance mismatch between data
and model development and operations,
blocking experimentations and reproducibility.
To illustrate how we dealt with all these challenges,
BrewKit was our zero friction, analytics framework.
This is built on a strong foundation
of Azure, Databricks and Delta.
BrewKit would be a unified analytics platform.
To reduce our impedance mismatch between access to data,
ability to perform data science and operationalization.
We wanted to make sure the smallest of teams
at Starbucks had the ability to do at scale data science
and data engineering with this framework.
BrewKit, our massive scale, data framework,
has all the necessary capabilities in a box, to perform
at scale data science and data engineering.
We leverage multiple larger services
and homegrown Starbucks engineering
to light up a functioning environment.
Multiple SKUs translate into data engineering
and data science templates all this generated declaratively,
in a matter of minutes.
We have services like Mlflow and Azure ML
that help our model management life cycles.
All this background storage services,
such as, Delta Lake use Azure services,
such as, Azure Keyboard to help us secure key management
across different environments.
From an infrastructure essential standpoint,
we have Databricks as a gore engine for on-demand compute
and the notebook collaborative experience
helps teams collaborate and experiment at a rapid fashion.
The ASO Glare is powered by services, such as,
Cosmos DB that help us enable internet scale,
low latency serving.
We also provide our commission and starter notebooks
and templates across a variety of use cases
so that users can bootstrap quickly onto the environment.
All activities are audited for governance reason
as well as, return on investment.
Moving on to our second challenge, data and ingestion.
And huge focus data for teams has been to ingest data
at scale using custom Spark Kubernetes with massive.
With inherent benefits of Delta,
such as, Azure transactions, metadata handling
and schema enforcement, we can use services
now, like Azure Event Hubs
and Spark Structured Streaming that help us
process millions of transactions, at scale, per second.
For long running pipelines, fault tolerance is trivial
now with structured streaming.
More importantly, Delta has now helped us
build out our historical data
and live aggregations together to make sure
we are now giving our store partners real time insights
on data, based on history and on current time.
Construed pipelines that hold transaction data
that are cleansed, audited and processed to be sent
right back to the stores, within minutes to ensure
that our partners have this latest data
to make decisions with.
The ability to perform stateful and stateless aggregations
helps us tailor trade offs based on use cases.
Features like compaction, auto optimization, help us
now have the right partisan keys in our partisan Delta Lake
to get us 50 to 100x performance gains
as well as storage optimization.
It's crucial for us to support time travel
and versioned data for data and model prominence.
Operations aren't Delta such as mergers of dates
and deletes help us built out
our own automatization frameworks.
We can also have large con separation on the Delta layer
with discovery and integration zone
where we can apply schema enforcement based
on the source of the data, as well as,
apply retention policies.
This process to a highly industrialized process
where we aggregate data across different zones,
and load data from a variety of sources.
Using the workspaces, clusters and job ABIs
and orchestration tools like Air Flow.
We have predefined patterns for these ingestion processes
that help us define retention policies
as well as, partition according to the use case.
Features Delta helps us with features
like add importancy with merge operations
and with fast changing change data capture use cases.
Time Travel provides snapshot isolation
for fast changing tables.
We also we go on, we make sure we are
now going in enterprise Delta Lake
with the right permissioning and access control.
Overall, the strategic view has been
here to commoditize data ingestion
to such an extent, so that, the teams can focus
on business problems up the value chain
rather than focusing on moving data from point a to point b.
On the consumption side, which is the third challenge
we had, we wanted to make sure your experience is
as good as, you walk into one of our premier experiences.
We sell over 50 plus workspaces on the enterprise
with different business units
with logical separations between them.
This gives us flexibility to monitor, secure
and segregate these environments based on the use case.
We also run our own version of our metadata
sync process across these 50 plus environments
to sync across these workspaces at scale.
So all users are looking at one version
of published data with a single source of truth.
The successful data enablement and BrewKit journey
has now led us to focus on our AI/ML strategy.
We have enabled convergence of data
and enabled BrewKit for the creation of models.
In the past, our goals were to deploy environments
in a matter of minutes, to deploy data pipelines
in a matter of minutes, which we have largely achieved.
Now our focus is now to democratize machine learning.
And how do we enable our data science disc
to enable machine learning models in a matter of minutes?
We have our framework AI Reserve, that we are
in the process of deploying with a goal
to achieve exactly that.
Powered by Mlflow and Azure Services,
AI Reserve will be guiding larger constructs
in our enterprise, including a model marketplace.
This will be used in many use cases across the board.
Such as, tour operations, quality of service analysis,
personalization experience and much more.
The ML stack comes together with out of the box
with our BrewKIt environments.
User have a plethora of environments
that they develop their code in.
We can use version control to deploy builds based
on integration with Auzure services like Azure DevOps.
Based on use cases like On Schedule or Drift Scenarios,
users can use Azure ML or Mlflow for
their model life cycle management.
And all this backed on a persistent data store
on the Delta Lake.
We also make sure IA Reserve integration allows
our data scientist to containerize their solutions
and use Azure Kubernetes Services to essentially
sell our REST APIs which leads to the larger construct
of the AI Reserve model marketplace.
To release realtime data along with third party data,
is driving our decisions across the board,
including store reopening strategies.
All this is being supported by a unified data
and AI Solution.
We now have petabytes of data located
on Delta Lake at massive scale.
Hundreds of data products, built on Spark.
An average deployment time of pipelines
and environments in minutes.
Over a 1000 plus pipelines across engineering
and user workspaces.
And more importantly, these are all highly reliable
and fault tolerant.
From a data team collaboration and productivity standpoint,
this has been huge.
The tooling is collaborative.
We also now foster a culture of experimentation
and self-service and maintain shared responsibility
across our environments and the focus is now
on outcomes and our technology or infrastructure.
I'll encourage everyone to check
our additional talks with Starbucks at the summit.
Including how we operationalize our big data pipelines
as well as our machine learning.
Please be sure to visit www.stories.starbucks.com
for latest news and initiatives.
I hope you have a great SPARK+AI SUMMIT 2020.
Thank you very much.