DINA GRAVES PORTMAN: Hi, Martin.
I'm working on a design doc.
Do you have a few minutes to help me with it?
MARTIN OMANDER: Yeah.
I love design sessions.
DINA GRAVES PORTMAN: Great.
OK.
We have these on-prem servers and our new developers
keep building new things, which is great, but I'm thinking I
need some kind of web application
to help me manage it all, that and some batch
processing to keep track of the state of things.
MARTIN OMANDER: OK.
This new application sounds like a good use case for serverless.
Let's get started.
[MUSIC PLAYING]
DINA GRAVES PORTMAN: High level, we
want to make sure they go through the proper approval
process before launching a new service,
and frankly, it's getting difficult to keep
track of everything.
I want to build an application that
helps me manage open ports on the servers,
keeps a log for our auditors, and allows developers
to easily file requests for opening up
access for new applications.
We have three distinct users.
We have a developer, an admin, and an auditor,
and the developer should be able to make requests
for new open ports through a web application,
and the admin should be able to approve those requests.
The admin should also get notified
if there are ever any open ports that he did not approve.
And then the auditor should be able to see
just the entire logs of all the things that
have been opened at any time.
MARTIN OMANDER: Oh, cool.
Sounds good.
So we have them here.
We started our system diagram, and down in the lower right
hand corner, we put the servers that they're monitoring.
This is a company internal app.
Is that right?
DINA GRAVES PORTMAN: Yeah, it's completely internal.
MARTIN OMANDER: Oh, got it.
So they are employees.
They should be able to reach it, but no other external users
should be able to reach this application?
DINA GRAVES PORTMAN: Exactly.
MARTIN OMANDER: What might we put in there
to make it easy to keep employees in and everybody
else out?
DINA GRAVES PORTMAN: I think another thing to consider
is, first of all, it's only internal employees,
but also, these two employees have different permissions
of different roles of the things that they can do.
So it should be something that allows us to kind of tweak
and tune the permissions levels.
MARTIN OMANDER: Oh, that's a good point.
Developers and admins can do different things, yeah.
So I'm thinking, then, identity aware
proxy here backed by App Engine.
It's really easy to put IAP and App Engine together.
Do you think that would work here?
DINA GRAVES PORTMAN: Yeah.
No, I think that would work great.
OK, so then App Engine with a little IAP in front of it,
controls, like, who can get in.
OK, so now we need to think about storage because they
are going to be making requests and approving requests,
so that means that we need to be able to create, read, update,
delete some kind of something.
And to me, the big question is, do we use NoSQL or SQL?
MARTIN OMANDER: So we're trying to keep it serverless, which
would mean NoSQL, and I'm also thinking
there won't be too many reports of-- we
won't do too many aggregations, like averages, and sums,
and stuff like that, right?
DINA GRAVES PORTMAN: I think just
thinking about how the people are interacting with the data
and also how I would model the data,
NoSQL is really feeling like the right direction
because they're not going to have to do big reports.
And even when you think about, like, we have a server,
the server is represented by the IP address
and then we have a set of ports underneath it.
So if I were to have a document in a NoSQL database,
the document, the ID would be the IP address,
and then we'd have the list of ports in it as rows and then
whether or not it should be open.
MARTIN OMANDER: Oh, yeah.
It's kind of hierarchical like that, yeah.
I agree.
So yeah, let's go for NoSQL.
Firestore is great and serverless,
so it would go very well together
with the other serverless components here.
Each document, I guess, would be a server.
And as you said, underneath it has ports, and permissions,
and things.
DINA GRAVES PORTMAN: For our auditor
and also for just our peace of mind,
we want to write a little port scanner so that we can check,
we can check that all the ports that are open
are the ones that have been approved,
and I want this to happen nightly,
and I don't want any interaction.
MARTIN OMANDER: OK, so nobody goes in and clicks and says,
scan the ports right now.
OK, got it.
DINA GRAVES PORTMAN: Yeah, like a cron job.
But I don't want the cron job on my computer.
MARTIN OMANDER: Yeah, yeah, yeah.
And Dina closes her laptop and goes on vacation
and no cron jobs are running.
Yeah, so right.
And, of course, the way to run cron jobs in Google Cloud
platform is Cloud Scheduler.
I think that would be a great choice here.
We can set it to run nightly, or change it to run hourly,
or whatever.
And then the Cloud Scheduler would need to call something.
What kind of component should it call, I wonder?
DINA GRAVES PORTMAN: So thinking of the functionality
that we want, what we want to do is
we want to have the IP address and we
want to attempt to make a request
to connect to every single port on the IP address.
So we're thinking about, like, nested for loops, which
actually sounds really scary.
Oh my gosh.
But the first thing that we need to do
is we need to get the list of IP addresses.
So maybe we can break up this functionality a little bit
and have the first function just call to Firestore
and get that list of IP address.
MARTIN OMANDER: Oh, I really like that.
Yeah, so OK, so that first function is really simple.
Like you said, it just gets all the records
and does no port scanning on itself, OK.
So no nested for loop here, but that one
is just pulling out the data.
Now, who does the actual work?
DINA GRAVES PORTMAN: That is a really great question
because I don't want to just call
another function because then we have
the problem of the nested for loops.
And if you're going through the tens of thousands
of maybe millions of ports, like,
something's going to break, and it's
going to take a really long time, and then
what are you going to do, just restart the whole thing?
So--
MARTIN OMANDER: Yeah.
Right.
DINA GRAVES PORTMAN: I'd really like
to do some kind of asynchronous, fan out, kind of something--
MARTIN OMANDER: Oh, you said asynchronous.
When you say asynchronous, Dina, that makes me think of Pub/Sub.
So would it work if the orchestration function
published one Pub/Sub message for every server
and then we had those messages trigger different Cloud
Function that tested just one server each?
DINA GRAVES PORTMAN: That is so good.
That's, like, even better than what I was originally
thinking because what that allows us to do,
then, is if our function fails for some reason, right,
then we only have to retry that one port on that one server
and we don't have to worry about the hundreds of thousands
or millions of other ports.
It's just the one.
MARTIN OMANDER: Yeah, right?
So it doesn't get stuck, no time outs.
OK.
This is really good.
So the port tester, then, gets just one server
that did your test or maybe one server and one port
combination.
It tests that, so it's a really simple function.
Now, it needs to put the data somewhere,
and I'm especially looking over there
to the left at where it says Admin.
The admin should get an email whenever a port is open
that shouldn't be open, right?
How do we get that data over there into that email?
DINA GRAVES PORTMAN: Right.
So I think, then, what we need is
when we get to this port tester Cloud Function,
we're feeding in the IP address, the port,
and whether or not it should be open,
and then we're receiving back the response of,
like, true or false, it was open.
And so then we want all of these four pieces of data to--
especially if it doesn't match, like, if it's open
and it shouldn't be open or if it's not open
and it should be open, because that can happen too,
then we want that to trigger an email.
But also, we want everything, not just
the things that don't match, we want everything
to go to the auditor.
And so I think maybe we're going to do, like, another Pub/Sub
thing.
MARTIN OMANDER: Oh, right.
Because-- and I think the reason that this fits very well
is when you said this data goes to two places,
Pub/Sub is amazing for that, right?
You publish it once and then you can have multiple subscribers.
DINA GRAVES PORTMAN: Yes.
Yes, multiple subscribers, and we
can do a filtered subscription so then our little function
doesn't have to do any logic.
It doesn't have to say, like, hey, if else,
just go here, go here.
We're just sending everything to Pub/Sub
and then we can have our filtered subscription that
will pick up whenever the match thing doesn't happen, you know?
And that can trigger an email.
MARTIN OMANDER: Oh, right, right.
Yep, yep.
So we have that filter subscription
could go into App Engine.
App Engine has that little logic, little handler in there,
that whenever it gets one of these messages,
it does what you just described.
It calls an email, API sends an email.
Cool.
DINA GRAVES PORTMAN: I love it.
MARTIN OMANDER: So we have that dangling thing down there.
The auditor looks a little lonely in the lower left hand
corner.
How do we get data to them?
DINA GRAVES PORTMAN: Right.
OK, so let's work backwards a little bit.
MARTIN OMANDER: Yes.
DINA GRAVES PORTMAN: The auditor is
going to want to look at a report,
maybe, like, in a dashboard.
And when I think about dashboards,
I start thinking about SQL.
Like, you want to be able to write queries.
So we should be storing this data
in something that has kind of like a SQL interface.
And I'm thinking because we're just, like, appending rows,
and it's going to be a lot of data,
I'm thinking BigQuery might be a good option.
MARTIN OMANDER: I really like that, yeah.
Yeah, because this is exactly what BigQuery is for, right?
Reporting.
And yeah, so we just put a little Cloud Function in there
that is triggered.
So I guess this one would not be a filtered subscription.
It would be a regular old subscription, right?
DINA GRAVES PORTMAN: Yeah.
Yeah, everything.
We want everything.
The auditor wants to see it all.
MARTIN OMANDER: Cool.
All right.
Very nice.
So now we have auditor wants a web page,
the data is in BigQuery already.
This is great.
Now-- but we need that web page and BigQuery
to meet here somewhere.
I mean, should we sit down and write the web app
and figure out some library that can draw graphs and things?
DINA GRAVES PORTMAN: Oh, god.
No, I don't want to do that.
MARTIN OMANDER: I've done it on so many projects,
I really don't want to do it either, yes.
DINA GRAVES PORTMAN: OK.
I'm going to throw this out there.
I think we should use Data Studio because when you're
looking at BigQuery, there's already
a button that's like, hey, look at this data in Data Studio,
so there we go.
Easy .
MARTIN OMANDER: I the way you think because now a business
analyst could draw the charts in Data Studio
and a developer can focus on the other pieces of this.
And yeah, no more building yet another web
app that draws charts.
Somebody has already done it for us.
DINA GRAVES PORTMAN: Awesome.
This looks so good.
OK, let me just walk through it real quick, make sure I have it
all.
I think I have enough for my design doc,
but let's go through it.
OK, so for our web app, we're going to have App Engine
and we're going to have Identity Aware Proxy IAP,
to control our permissions, and we're
going to store all the requests in Firestore.
And then every night for our audit,
we are going to have Cloud Scheduler basically kick off
this pipeline of Pub/Sub and functions that ultimately
is going to scan all of our ports,
and we're going to write all of the results to Pub/Sub,
and we're going to have a filtered subscription that
sends the bad results back to App Engine,
we're going to trigger an email to our admin,
and then we have all of our results going into BigQuery,
which you are going to show the reports in Data Studio.
MARTIN OMANDER: Yeah, I really like it.
I think this flow would work well.
This would actually-- this would be a fun application to build.
DINA GRAVES PORTMAN: Yeah, I want
to start building this Cloud Scheduler thing right now.
It looks awesome.
MARTIN OMANDER: Yeah.
Yeah, my brain is tingling.
Awesome.
I love these design sessions.
Thank you everyone, for watching.
This is how you might build serverless-based systems.
Everything you saw here was serverless.
If you have any questions for Dina or me about anything
you saw in this video or if you have requests for us
for other types of applications you'd
like to see us do design sessions for, please add them
in the comments below.
Until next time, bye.
[MUSIC PLAYING]