[MUSIC PLAYING]
PRIYANKA VERGADIA: Have you ever wondered
how video monitoring systems are architected?
Well, today you're in luck as we explore
how Arcules, a video monitoring IoT company,
created their architecture in the cloud.
I've invited Ben, their cloud and security architect, who
will share more with us today.
Hey, Ben.
Welcome to the show.
How are you?
And what do you do at Arcules?
PRIYANKA VERGADIA: Hi.
Thank you, Priyanka.
My name is Ben.
I am the cloud and security architect,
although Arcules is still a startup vibe.
We operate under more of a collaborative product
team, which allows me to wear multiple hats.
I not only get to work on architecture and security,
but also get to work on machine learning,
look after multiple teams, and also get
to experience more hands-on with our enterprise customers.
PRIYANKA VERGADIA: Wow, that is all sorts of different things!
From ML to security, that's a huge spectrum
that you work with.
So we know, at a very high-level,
how video and IoT monitoring works,
but I'm curious what Arcules exactly does.
BEN: Arcules takes all of the information
from a multitude of devices at the edge.
And what we do is we have a small appliance
at the edge that captures all that information
and then processes it and then pushes it
all the way through into the cloud, specifically the Google
Cloud.
PRIYANKA VERGADIA: All right.
That makes sense to me.
We have some devices.
They're sending data to the cloud,
and then there's some processing happening with that data.
How is it actually architected?
And, I guess, before we even get to how is
it actually architected, what is video security as a service?
And then we can dive into the architecture.
BEN: Sure.
Video security as a service is like a SaaS-based product
that's obviously storing video.
Typically, our traditional competitors
are VMSes, which are video management solutions.
What they do is they collect all this video information,
make it easy for people to find incidences or people
within video across hundreds of cameras.
PRIYANKA VERGADIA: OK, Ben.
So what are the challenges in building a system like this?
And I'm actually really curious about why you
chose Google Cloud for this.
BEN: We chose Google Cloud primarily because of speed.
We evaluated multiple cloud providers at the time
because we wanted more of that on-premise feel.
Google won out.
The other key aspect to this as well
was access your product managers and engineers directly.
That really helped us as a startup
to be able to get more information faster and help
train our engineers up quicker as well.
PRIYANKA VERGADIA: All right.
So time to get into the architecture and the details.
And you've given an indication that obviously there's
customer sites where you have the IoT devices that
are sending the data into Google Cloud
and then there's processing happening on there.
But I'm curious about how the data makes it
from those IoT devices into the cloud and then
what happens afterwards?
BEN: So each of our customer sites
ultimately have a small appliance
at the edge that allows us to buffer
the information before it makes its way to the cloud.
As we all know, internet's super reliable.
So we are hardware-agnostic.
We can deploy on any Bare Metal.
We can also deploy inside of VMware.
And then we have Kubernetes at the edge.
That allows us to deploy multiple containers
and different systems as needed and, ultimately, shift
compute where it should be.
Then, also, we have our last-mile connectivity.
That's really powerful because that's primarily
the reason why we chose Google.
We're able to provide almost near zero latency.
Ultimately, it's just that last mile piece.
So we're able to actually deliver
video subsecond to anyone, anywhere in the world.
PRIYANKA VERGADIA: Awesome.
So we got into Google Cloud at this point.
Before the data gets into the services running on GKE,
you must have some load balancing
that's happening, right?
BEN: Yeah.
So we actually have over 90 micro-services in GKE today.
So we really utilize GKE.
It's super awesome.
But before, obviously, we hit the GK instance,
we do use our load balances.
We use both L7 load balancers.
We do utilize some of the more interesting features
that you have inside of the HTTP, HTTPS load balancing
and endpoint connections, but we also
use a lot of L4, for instance, with WebRTC.
And then, finally, we have IoT Core as well.
With IoT Core, we actually use that for some of our metadata.
The metadata flows up through from some
of those services at the edge.
And that feeds into pub/sub, which
then also feeds into BigQuery.
PRIYANKA VERGADIA: We didn't talk much about the databases
that you're using.
I assume you have pretty much the spread of both relational
and non-relational in this use case.
But can you talk more about the databases?
BEN: Sure.
We utilize all kinds of databases,
some hyper-specific, some not so specific.
Probably one of our biggest uses today, we actually
use Cloud SQL, again, with that lighter weight,
not needing to have to manage those instances.
We use both Postgres and a bit of MySQL, primarily Postgres.
We also use, like you mentioned, a couple
of other more exotic databases.
We actually use graph databases, including Arango.
We actually have an interesting NIST-based permission graph
model which makes permissions way better
for horizontally scaling.
Traditional methods don't work so well.
Singlestore we actually use that for a high rate of ingestion.
We've tested that to more than 400,000 inserts per second.
The Postgres is mostly used for our device configuration.
We have tens of thousands of devices globally.
All of our configurations are stored in our Cloud SQL
Postgres units.
|And then BigQuery we actually use for slow storage.
So as we connect and have all these different pieces
of information coming in through metadata,
we slow store that inside of BigQuery and obviously really
powerful to be able to run jobs across large data sets.
PRIYANKA VERGADIA: Wow, that is a lot.
Like I imagine, it is a spread of all sorts of databases.
I want to go back a little bit and dive into the video files
themselves.
I think you mentioned, briefly touched on the fact,
that these video files are chunked,
and then you put them into cloud storage.
So can you talk a little bit about that process?
Because, yes, the files are stored
in cloud storage, but the way you
do them sounds really unique.
So I want to know more about that.
BEN: So the video is mostly stored today
inside of the custom container format that we ultimately use.
Under the hood is ultimtely H.264, which
is a very common video format.
Within video, you have keyframes,
and those key frames are chunked up.
Depending on the camera, they may
be at one second, two seconds, four seconds.
And that's how we ultimately split that data up.
So we could be pushing millions of files per month per device
depending on how that's created.
We actually recently built a new way
of aggregating that information, so
that we didn't have to spend so much money on to PUT requests.
And we use that in combination with pub/sub.
All these messages fly through pub/sub
and allow us to get those into the information,
into the databases to allow quicker access.
PRIYANKA VERGADIA: Well, that makes a lot of sense.
Now, deployment, what's your CICD like?
How do you deploy these microservices?
I heard 19 services, so I'm really curious how
the deployment works.
BEN: We use a multitude of different services.
Deployment itself is typically through Seneca,
but our build process is actually through the Cloud
Build platform.
We also heavy users of the container
services that you have as well for container management
and deployment.
PRIYANKA VERGADIA: So how do you actually monitor these systems?
Because there is a lot happening here.
BEN: We use Cloud Operations primarily
as it's native to the Google platform.
That's a huge win for us.
We also use New Relic.
We also use Dynatrace for some services.
And then we have our own stack with [? Profoner ?]
and Prometheus as well, depending on the services
that are being monitored.
PRIYANKA VERGADIA: When it comes to dealing
with a platform like this, security obviously
is extremely important, because you're
pretty much doing video and dealing with sensitive data.
So specifically monitoring any open ports also,
it could be a problem.
So how do you make sure your architecture and components are
secure?
BEN: So that's really, really interesting.
We use, again, multiple products to do that.
We have our third-party penetration test companies
that we have, we contract with to make
sure we have that covered from an external, true external
perspective.
Internally, we do utilize the API scanners.
We also use the Cloud Security Command Center
that you have as well inside of the product that
does provide some pretty good basic CIS stuff, which really
helps us understand which ports are open, whether something
was configured.
We do have some other special sauce as well,
which I unfortunately cannot talk about.
PRIYANKA VERGADIA: Thanks, Ben.
I learned a lot with that entire architecture diagram.
But what do future technical enhancements
to this architecture look like?
And what is the next generation of video analysis system?
I'm pretty sure you're imagining it,
so I want to know more about that.
BEN: From an architectural standpoint
and from more of like kind of operation standpoint,
I think the next things for us are actually
implementing Istio, maybe Jaeger as well, getting
more distributed tracing in.
From a machine learning perspective,
we're focusing on machine learning operations pipelines,
allowing our users potentially be
able to retrain on their own data
and getting much more granular in that aspect.
And then, I think that's the two key parts.
PRIYANKA VERGADIA: Thank you so much, Ben,
for spending some time with me today and sharing
about the video monitoring systems.
BEN: Thank you, Priyanka.
It's been awesome.
PRIYANKA VERGADIA: I don't know about you,
but I learned a lot today--
how IoT devices send data, how video monitoring is done,
and how all the data is used for business insights and machine
learning.
My mind is absolutely blown.
But the cool part is it all begins with a plan,
and there's always a really cool architecture to uncover.
I've included some resources for you
to check out in the description below,
and I'll see you next time.