Welcome to, 'This is My Architecture.'
My name is Ran, and joining me today, is Ariel from Sisense.
- Hi Ariel. - Hi Ran.
Ah, before we dive into this, tell us a bit about Sisense.
Sisense is a BI company... allowing you to see the data that you have.
Ah... that is coming from different sources.
Understanding the data, modelizing the data,
and getting insights from that data, while maintaining the overall architecture.
OK. So, you have BI analysts going and using that dashboard,
having all kinds of different data in it.
Um... they need to configure those different data sources, in order to see them.
How does that work behind the scenes?
So, you have the users, that are seeing that information.
OK? Different type of graph.
But, a lot of data comes from different types of sources.
OK? Whether those sources are in the customer side,
or web sources or different locations.
So, that can be, just a regular database, inside the organization,
that you want to get in, inside your Kubernetes clusters.
Or that can be a Redshift that's running in the same VPC,
that you want to get in, that information.
You have the S3 buckets, that you want to expose by Athena.
And, you want to expose that, coming also, into your overall data.
So, you want to digest that data, OK? To manipulate that data.
Understanding, that sometimes, it's a spelling mistake on the data.
You want to say that, the Redshift column, EKS,
is the same as column EKS over here.
OK? They're coming from different sources.
So, you want to mesh all the data together,
and, allowing the user, at the end of the day,
getting the graph that this comes from-- this graph comes from here, here and here.
Doing that, you need the Sisense, all the ETL processes,
that digest the data and keep it in a shell-storage solution.
OK. So, as a shell-storage solution, you're using FSx for Lustre?
Ah, tell us a little bit about why you chose FSx for Lustre?
As you see, that we have a lot of ingest comes in.
OK? A lot of manipulation comes in.
A lot of that data needs to be kept... ... in a very fast way.
A lot of our customers grab the data, sometimes in every hour,
sometimes every five minutes, and digest all the data.
And we need a highly available solution,
that can be very, very fast in responsiveness for keeping the data.
Another process, that gets the data from the shell storage,
and be able to answer the different clients,
on complex queries that happens in a millisecond way.
The way that we do that, is, we form that query service,
comes into the different shell storage area,
memory-marked files, we manipulate the data,
and get the responses to the different type of clients.
OK. So you can ingest the data at the backend, either to a FSx for Lustre file system,
or to Redshift? How do you decide what to use and when?
So, even if the data is kind of a hot data,
that you need a lot of responses, very frequent, in a milliseconds way.
We use our ElastiCube logic, that is In-Memory Database Column Store.
OK? That mount directly from FSx for Lustre.
If the data is more as a warm data, massive number, size of data,
terabyte of data, we use the Redshift as the warehouse database.
And, we can query directly to the Redshift.
OK. So these workers are running inside containers on EKS clusters?
So, why did you choose containers in EKS for this workload?
We get a lot of automatic high availability, scale, resiliency, load balancing,
coming from the Kubernetes clusters.
Every workload that we do, can be, sometimes small, can be huge,
and take a lot of resources.
Sometimes a lot of operations happen in parallel for the ingest,
and also for the query, each one has its own timing,
and its own workload. OK?
During the day, comparing doing at night, so we do all kinds of manipulation.
In order to do that, in a very dynamically scalable solution,
we utilize Kubernetes, the auto-scalers of the parts,
auto-scalers of the nodes, everything is automatically scaled.
Kubernetes sees the workload and can inject additional compute as needed.
The kind of-- be able to talk about big-data analytics,
while having automatically scaling, that, we're gaining from the FSx for Lustre,
because we automatic-- very fast mount from the FSx for Lustre,
and can serve the data faster.
Great, so in this case, why did you choose FSx for Lustre,
and not some other type of file system?
The additional capability for the FSx-- so first, FSx for Lustre,
is very, very fast responses, low latencies, a lot of throughputs.
But, it also has capability to keep the data in the S3 bucket.
So if you need a resiliency to bring up different systems,
you can just bring the system from that bucket.
and you can back up the buckets, and you can replicate that.
So, it's a very-- also efficient for disaster recovery.
Great. Sounds good!
So, it sounds like you built a really resilient, fault tolerant
also highly performative auto-scaling system.
Thank you very much for sharing.
Thank you Ran.
And, thank you for watching, 'This is My Architecture.'