Welcome to 'This is My Architecture'. Today, I have here with me Ido from Mobileye.
Hi Ido. Hi Boaz.
Ido, tell us a bit about Mobileye.
Mobileye is a world leader company in camera based driver assistance systems
with over 70 million Mobileyes inside vehicles around the world.
It is also a world leader company in autonomous driving systems.
In addition, Mobileye is providing a full suit of data services,
using an award-winning roadmap in technology and vehicles.
And this is the team you're coming from. Tell us how you do that.
To do that, we collect anonymized data using sensors and visual inspections from cars.
This data is being sent to a REST API in a different AWS account,
and then stored into S3 for future usage.
Okay, so cars are driving around the world collecting data storing into an S3 bucket, then what?
For that purpose exactly, our team developed Dynamic REM.
Dynamic REM is a platform, which allows us to provide customers
with mobility intelligence and live events data.
It is composed of three main layers: the process layer,
which is responsible for processing the data received from the cars, and then store it into the store layer.
The store layer contains multiple technologies in order to support system different needs.
It is composed of S3, Elasticsearch and CockroachDB.
Then platform insights are exposed to customers using the consume layer.
Okay, where is the typical customer or the system?
So Dynamic REM has multiple customer types, for example: roads and cities, planners and operators.
So let's dive into the architecture. So you collected the data,
move it to Dynamic REM, what do you do with this data?
So as I mentioned before, data is stored in S3, from there it is being pulled to SQS.
And then being processed by EKS workers, those EKS workers
actually scale automatically, according to the relevant SQS queue size
and can scale up to 800% during the day.
At the next phase, we pre-process format,
store the data in S3, and then send it to a set of algorithms
implemented by Step Functions executing multiple Lambda functions.
Can you give you an example for one of those functions? Yes, of course, Boaz.
So, one example for an algorithm can be the extraction
of potential construction zones from vehicles.
Okay, then what?
Then, the results are being exposed to the rest of the systems
components, for example, a sophisticated clustering mechanism,
which combines observations from different vehicles into clustered events.
Another example can be there are aggregation mechanism,
which aggregates data by time and road segments.
Cool, so you already collected some data, got some insights, what are you doing with these insights?
So, for example, if I'm a city planner, the platform can provide me
accurate hourly data regarding pedestrian and bicycle traffic,
which allows me to plan better roads and cities.
Okay, and how can I get this data from the system.
So after the data was calculated,
we save it into Elasticsearch and CockroachDB.
And the next phase, we serve this data to our customers using an EKS cluster.
This EKS cluster is serving a secured REST API
and web application, which our users
are using to consume the data.
Okay, so I can actually get the insights from this.
I wonder how many events the system can process.
So today, we use between 100 to 800, EKS workers
in order to process around 10 million observations per day.
Okay,10 million observation per day doesn't sound much like you have 70 million cars driving all over the world,
the amount of data each car can produce. Why is that?
That's an excellent question, Boaz, we actively limit the amount of data flowing into the system
in order to control costs and allow a per use model.
Okay, so I guess this is obviously intentional.
Okay, so let's talk about the evolution of the system. So this is the current architecture,
I assume we did not start with this exact architecture, walk us through the evolution of this architecture.
Sure, so first of all, we believe that evolution is a crucial step
in building large scale systems.
We actually started with Spark streaming, EMR and Kafka.
None of them on the board. Yeah.
And the next phase, we switched to Lambda functions
in order to gain serverless, highly available architecture.
Okay. At the next phase, we moved most of the Lambda
into EKS workers, because when a certain scale was reached
easy to compute on spots was much more cost-effective.
That's a great point, so when you have a large scale, this is something you should consider. Exactly.
I see a lot of different technologies in the process layer,
you have Lambda functions, you have EKS, Kubernetes in general, why is that?
So first, we believe the different solution requires different architecture and services.
And we're not afraid to combine those into a single platform.
Our algorithm execution flow requires parallelism and flow management.
Step Functions is a service, which provides exactly that for us.
Yeah, makes a lot of sense. Last question regarding the evolution,
let's talk for a second about the store layer. So using a lot of different services,
S3, Elasticsearch, CockroachDB, which I never heard about and never used it before.
Why do you need so many? So as in before evolution,
so we actually started with RDS
on RDS Postgres on top of AWS.
Then, due to scale and performance requirements, we switched to RIDB.
At the next phase, we chose Elasticsearch because we searched for a database providing document based database,
which allows self indexing, and it is a database which is cost-effective.
Okay, CockroachDB. So recently, we have found out that Elasticsearch
is not the best solution for conducting frequent updates.
And we moved some of our components to use CockroachDB.
CockroachDB is a relational database, which can support frequent updates to tables.
We deployed CockroachDB in an EKS cluster,
which allows us to enjoy scalability, and availability.
Okay, and obviously, a lot of components, a lot of moving parts,
trying to keep the same architecture using EKS in a lot of places.
I see this is a live system. This is a very evolving system.
I wonder what are the future plans for Dynamic REM.
So we have a long list of future plans, let's try to talk about two of them.
First one is related to image processing. So today, most of the image processing
and visual inspection actually happens in the car.
We would like to enable cloud image processing. To do that, we're going to enable
our pre developed image processing engine.
This engine allows us to process images in stream and gain the result.
We are going to deploy this engine on top of EKS, but this time with GPU instances.
Yeah, the technology, great. Yeah.
Another item in our to-do list is related to cost reduction.
So Dynamic REM platform is going to grow, and data flow will keep growing.
In order to do that, we must make sure that we maintain a solid and reasonable price level request for vehicles.
Absolutely and there are so many so this is absolutely crucial.
Ido, thank you very much for sharing. Thank you Boaz.
And thank you for watching 'This is My Architecture'.