- Hi, I'm Daniel Tibble.
I'm the head of data at Wejo,
but I'm gonna talk today
about how we're using the power Databricks
to make petabyte scale data
consumable to all of our customers.
So at Wejo, we take data from connected car vehicles,
so vehicles are connected to internet.
At the moment in the US,
we've got over 15 million vehicles,
and we're streaming that data in near real-time,
so we take the data from the vehicle into our engine
and make it consumable for our partners
in quite often less than 40 to 60 seconds
and it's huge amounts of data,
you know 18 billion data points a day,
900,000 per second data points.
Now, this is more data than you know, Visa processing.
This is more data than,
you know most people are used to dealing with
and we've got over five petabytes
and that's probably actually closer to six right now
because it's growing all the time.
So you know, it's a really huge amount of data
that we're dealing with at Wejo.
So at Wejo, we're making all of that data available
to a large number of consumers,
so we work in the traffic management space,
we work in the retail space,
we work in insights across all different sectors
and what we're doing is we're taking that data
and making it consumable
for a number of different product types.
So we've got a livestream so we can stream
all of those billions of data points
or you know a subsection of for specific use cases.
We might be looking at real-time parking
or real-time traffic management.
We can transfer the large data in batch
via technologies on the web such as S3.
We're making more and more intelligence-based products,
which customizable reports
that aggregate the large amounts of data
around specific use cases and insights
and what what I'm gonna talk to about today, largely,
is a new offering we've got,
which is Mobility Intelligence Studio.
Now this is a self-serve data science platform
that we're levering Databricks for,
which is, it's making all of our data available
to a whole different type of consumer.
We don't necessarily have the infrastructure
or the capabilities to process that amount of raw data,
and this is really what I wanna talk about today
and also we've got some Developer Labs offering,
which is APIs, which you know you'll have seen
some of the things out there in the market
that you can subscribe to and you can call
and integrate with your own applications.
Just gonna talk a bit about the relationship
with Wejo and Databricks.
Now, we've been using Databricks for around 18 months now.
It's our primary analytical work,
our primary analytical platform,
and our data scientists, data engineers
and data analysts do 70, 80% of the work in Databricks.
Now as I mentioned before,
a large reason for that
is because of the size of the data we're dealing with
when we were looking at other platforms,
just the capabilities weren't there to be able to,
not just deal with the data
but give us the fully functioning data science environment
that we can get out of Databricks.
We've recently been extending Delta Lake
within our environment,
just to take advantage of the ability to you know, optimize
not just against your partitions
but re-indexing and some of the data lineage tracking
that you get in there is able to help us
in the CCPA and GDPR work areas.
Recently though, Databricks introduced
the White label functionality in their E2 offering,
and this is something that we've been exploring
in a number of different ways,
which is, first of all allowing a sample preview,
but also it's a way for us
to deliver intelligence and insights to our customers
and further, we're gonna look
at offering platforms the service,
because we got so much unique capabilities now
with our dealing with connected vehicle data that we can,
you know we're working
with some quite large vehicle manufacturers
to take advantage of that
and allow them to kind of develop their own platform
using Databricks and the Wejo intelligence.
So as I said before, we've got a huge amount of data,
and it's what makes Wejo unique,
you know, we've got the largest connected vehicle data
service of its kind anywhere in the world,
but it's also a largest challenge because we wanna make
that data available to solve use cases outside of businesses
that would traditionally have big data environments
and be able to process you know, petabyte scale data.
So we wanna open up to more users,
but we also wanna reduce the technical barrier to entry,
you know, we've got a lot of insight
that can be very consumable now and the investment
of being able to take this amount of data,
you know, it's quite long and it's got long lead times.
We've also got challenges around data privacy you know,
you guys in the US are a little bit further behind
in terms of privacy but in the UK, we've been dealing
with GDPR for quite a while now
and what we want to be able to do is make sure that people
can consume the insights and get the value out of Wejo data,
but also be able to control the data
within our own environment.
So I'm gonna talk about two use cases which speak to that.
Now, the first one is quite a simple one so it's a way
of being able to use the white label functionality
as a preview environment so that people can understand
how good and you know, the high quality
and the high coverage of Wejo data quite early
on in the sales cycle and then the second one
is much more about a platform so being able to kind
of give a subscription to customers so they can interact
and consume the Wejo data as the need to,
for that specific use cases.
So use case one is around evaluating Wejo data now,
you know, we've got some quite big customers
who are really, really wanna get the hands on our data,
but actually what that means is it's quite a lengthy
pre-sales cycle so previously what people had to do
is we had to enter into a legally binding agreement
so we could transfer the large amounts of data we've got
to the customer and that was you know, just sort of can even
get sense of the high quality it was outside of
what we were able to show them in our sales collateral,
so you know, it's quite a long process
and it meant we actually let the data lose our,
leave our environment which you know,
we don't really want to be doing.
it's something we gonna obviously we want to control
the data as much as possible.
So, what we moved to was we introduced a concept called
data labs and what this does is
it allows the potential customers to enter into
a very limited agreement where they can use
the white label environment to query a sample of our data,
so a very small sample so a week of the data
and because we can use the full power
of the Databricks environment that can do any querying,
any data quality assessment, the ones and they can really
get an understanding of the specifics of Wejo data
for their specific use cases
'cause it's very difficult sometimes each customer
has got different use case to kind of,
you know, understand it and it was taken up a lot
of my team's time particularly in data science
and data analytics to start pulling together
specific use cases for every customer.
So what we've been able to do with a white label environment
is get the data into the hands of the customer.
The architecture is pretty simple
so we've got our core ADEPT platform
which is Wejo's own ADEPT platform and that's where we keep
the majority of our data.
We very simply, we just make a copy of the data
into an S3 location, it's a one-off sample,
it's static, it doesn't update and then we just use
a very simple white label cluster access model.
So the each Egress partner can go into the environment,
run their own queries within the notebook,
we've pre-compiled some notebooks for them
but they can run them on the queries that data scientists
and data analysts can really get the hands on the data
and evaluate and see actually the quality of the data
without feeling there's a barrier in between
and as I said before, the real beauty of this is,
no data leaves a Wejo environment
so there's no upfront investment from the Egress partner
and it means that we've got full control over our data.
So this is an example of how I assemble
preview white label environment, as you can see,
it's a fairly familiar Databricks landing screen,
and there is some customization available in that
but what's great about it is it's fully federated
for each individual customer so they can only,
you know, work within their own environment that they can't
see other people's work and it could only feed the data
that we make available but even on top of that you've got
your own cost and management policies and you've got
your own user conflict within there.
It's quite a great way to kind of manage this sort
of environment and I'll just share an example of a notebook,
now as I mentioned before you know, we've pre-written a lot
of these notebooks but we do let people write their own code
and create their own notebooks and share them exactly
as if it was their own Databricks environment,
so what we've done here, if I just switched to a bit
of a dashboard view, just freaking get a sense for some
of the visuals that we create for them.
Now, we'll just find out this is synthetic data
so it's not really our data,
it's just a synthetic version of it so there's no private
or personal data gonna be shown here,
and yeah so, you know, it's a fairly standard dashboard,
you'll be familiar with a lot of these
and we've just pre-compiled some of these just
to get people interested.
One of the big things that people are interested in
is where's the coverage of our data,
so we're, you know just by creating a few simple images
like this we can start showing,
you know, where all of our data is across the US
and as you can see it fairly broadly follows population map,
we've put a couple of images in there, just certain visuals,
just that can understand what the can do with the data
in terms of the Databricks environment 'cause a lot
of our data's geospatial in nature, as you would expect
from connected vehicles, it's really,
you know, mapping type visuals that people want to look at
and now you can see just a bit of a heat map
of a very small sample of our data across the US
but it'll go down to individual data points
so you know, here with some individual data points
across the streets of Manhattan just so people can get
again a feel for the coverage of their,
this is all synthetic as I mentioned before
but really you know, what people actually use this for
is they very quickly move out
with our pre-compiled notebooks
and just really get stuck in, you know,
I'm sure everyone on this, everyone who's watching this
the first thing they wanna do when they see a data set
is go in and run their own queries and we encourage that,
you know, it's really what this environment is about,
It means that people can build around queries
and be in Spark, be in Scala, be in SQL
and the data scientists can you know, understand it
in the specifics so that they use case you know,
'cause people want to use it for different things,
we might have some people who wanna use it
for parking purposes, people who wanna look
at traffic flows, people might wanna be origin destination,
Saturdays and people might be interested in the latency
and the capture rate and that's all available for them
using you know fairly simple SQL and Spark queries.
So the other use case that we're looking at
with the distinct white label and this is quite new to us
but it's very exciting, a lot of interest in this
is using it in what we've run
in Mobility Intelligence Studio, now this is really
about gain access to Wejo intelligence product
so it's all of the intelligence and the insight
that our data science and data analytics teams
have built up, you know, we really are the experts
at the end and we've connected vehicle data
and we can start adding insight and using our own unique IP
to build reports and insights and again it reduces the need
for big investment in big data platform,
big data science capabilities.
Yeah, architecture is very similar,
there's not a huge departure, you know we separated
from our, what we call our core ADEPT
but we've actually moved the aggregated data
into a Delta lakes to give us a bit more control over the,
you know the tables that people can access
and the views that people can access but also it allows us
to optimize aggregated data rather than just looking
at the raw data but other than that it's a very simple,
it's a very similar model so you know, the clusters
are unique to each instance and each authorized user
can only look at that specific white label environment,
so similar to the previous use case,
it gives us a lot of control over you know what people
can see and do within that environment.
So this is an example of our Mobility Intelligence Studio,
so this is Traffic intelligence, the Wejo product
which takes the connected vehicle data at road level
and aggregates it and summarizes it so that people
can understand what's happening
on their individual road networks
and this is a fairly simple analysis
of a specific toll road in Pennsylvania
and it was over COVID but it was also
while the huddle lane closure,
that's the wanting to understand the impact
of the lane closure on the journeys
on that specific road network and as you can see, you know,
it's fairly simple insight and it was really great
because we could take, you know the petabytes worth of data
that we've gotten in Wejo and just very simply summarize it
and make it accessible for people in charts and graphs
that are over over and drilled down and they can, you know,
go into this and they can write their own queries
and they can further interrogate the data
as and when needed but what we're also doing with this,
and I'm just gonna switch to a different notebook here is,
you know, we've got a lot of people who,
you know, engineering firms particularly who just want
to really understand what's happening
on a specific road network and that's the big use case
we find it for Mobility Intelligence Studio
so we've been working with a couple
of large engineering firms that really drill down
into the specific use cases and this is a really common one
so what they're interested in is,
you know, very limited area and some very high level
and detailed stats on the area so you know,
this is very simply here you put in a couple of dates,
you put in a polygon which you can generate quite simply
and run the notebook and you can generate all
of this insight and this means that business managers
who don't necessarily know SQL or Spark or Scala
can start getting the insight that they need,
so just by putting in that polygon, they're able to
you know see where the vehicles are across the road network.
We can, you know zoom in, we've got counts
by 15 minute intervals, now the sort of things
that people use as far as each really forbid support,
so they might have, you know, a bit coming up
on an interstate in Arkansas, anywhere in the US
and the wants to get an understanding of how busy it is?
Which roads are used?
What times of day it's there?
And that's actually quite expensive to do,
without this sort of data 'cause you've got to send people
and install special equipment but really he can very quickly
start getting counts of the individual road network,
using the Wejo data when you can extrapolate that's up
to get a really good sense of when it's busy
and which roads are being used and you know if anyone here
has ever been to the golden gate,
some of these numbers might be familiar to them,
then we've got some origin destination images
so this is you know, again it's a simple heat map,
it's all anonymized so it doesn't go to specific,
so you can't get any personal information out of here
but you can get a really good sense of you know,
how the specific road networks being used,
so this is people who went over the golden gate bridge
every four days and they could see where,
you know, how far we got people coming from Reno,
coming from North Carolina
and then they should try and quite gonna state that
and you can go down to the real specific so this is actually
within that same polygon where people parked,
so you know, we've got a cluster of parking here
that looks like a viewpoint there.
So how do you know, almost over 200 vehicles park
on a viewpoint over that time and then we can do that
by time of day and do the breakdown
and this is really valuable for you know,
these companies to be able to get this data
in their hands without having to know Spark, SQL
or anything like that,
and the real big advantage of this for us is that none
of this touches raw data so you know, we've hidden all
of the code and we've kind of put a barrier
between the raw, any data that might be private
or personally identifiable or could be reverse engineered.
It's just not exposed in this you know,
the Databricks environment builds the aggregates for them,
it makes the aggregate tables available for query
but all of the raw data is behind that barrier.
So in summary, you know, we've been using Databricks
as our primary data science and data analytics platforms
for 18 months, two years now,
and it's where we built up a huge amount of IP value
and insight around the connected vehicle data set
that we've got,
within new white label and E2 functionality,
It allowed us to open up that data
and insight to our partners and it meant that they haven't
had to invest in huge amounts of big data processing
but also it means it don't necessarily need huge amounts
of data science, data engineering capabilities
to engage with Wejo data and it's really reduced
the barrier to entry and what that's meant
is we've opened up a huge amount of market
and use cases who wouldn't have previously been able
to consume Wejo data and Wejo insights.
So I'm just gonna open up to any questions, thoughts
or comments from the audience now.