MARTIN OMANDER: Hi, Guillaume.
In a previous episode, Mete showed us
Pic-a-Daily, the cool picture sharing application
that you guys built together.
GUILLAUME LAFORGE: Hi, Martin.
Oh yeah.
That was a lot of fun to build it using all those Google Cloud
service technologies.
MARTIN OMANDER: Yeah.
And Mete told me that you've rebuilt the application
recently?
GUILLAUME LAFORGE: Well, yes.
The application used the choreography pattern
with loosely coupled microservices
communicating with the events.
But we felt it needed a bit of a refresh.
We decided to convert it to the orchestration pattern.
[MUSIC PLAYING]
MARTIN OMANDER: Orchestration pattern?
What do you mean?
GUILLAUME LAFORGE: The idea is to have
somehow a central brain, the orchestrator, that
orchestrates and calls all the services directly.
One pain point with the choreography
of even driven apps is that it's sometimes a bit complicated
to figure out what's going on.
An orchestrator gives high level view of your business processes
to track their execution.
In the loosely coupled scenario when something fails,
you usually have to hunt down through heaps
of uncorrelated logs and errors to see
which particular service failed and why.
But with an orchestrator, you can
see the execution of your workflow in the cloud console
and see the result of all the service calls at a glance.
MARTIN OMANDER: Oh.
So what did you change to help with that?
And tell me more about orchestration.
GUILLAUME LAFORGE: Google Cloud recently
released a new product called Workflows.
It's an orchestrator.
It allows developers to create workflows.
With a series of steps, you can create business processes
that orchestrate calls to several services.
And the workflow calls those services as simple web APIs.
So we gave Workflows a try to get a clearer
vision and understanding of the underlying business
processes of our app.
MARTIN OMANDER: Wow, that sounds exciting.
Can you tell us more about this new architecture
and how Workflows fit in the big picture?
GUILLAUME LAFORGE: Our app business logic
isn't very complicated.
But it can be drawn as a simple flowchart basically.
MARTIN OMANDER: OK, it looks like the Google Cloud Console
here.
The Workflow's UI shows both a YAML definition file as well as
the flowchart diagram of that workflow.
GUILLAUME LAFORGE: Yep.
Exactly.
You write the YAML definition on the left,
but you can immediately see the equivalent diagram
on the right.
MARTIN OMANDER: Cool.
Could you describe this particular workflow
a little bit?
GUILLAUME LAFORGE: Sure.
When a new picture is uploaded in Cloud Storage,
or an existing one is deleted, an event
is fired, one for file creation, one for file deletion.
Those events trigger two functions
for those two situations.
But both start our workflow execution.
MARTIN OMANDER: OK.
That's how it starts.
What happens next?
GUILLAUME LAFORGE: If the file event isn't recognized,
we'll go straight to the end of the workflow.
If the file event is a deletion, we
remove the thumbnail pictures from Cloud Storage
as well as the picture metadata in Firestore.
If it's a file creation event, we
analyze the picture with machine learning from the Vision API
and extract information about the labels, the color,
whether it's safe or not to show the picture.
If it's a picture safe to show publicly,
we store the metadata in Firestore,
we create the thumbnail, and the collage.
MARTIN OMANDER: Ah, I see.
So you're saying that it's the workflow that calls those APIs?
GUILLAUME LAFORGE: Yes.
Calling an API is really just a call declaration
in the YAML definition.
Notice for example at the bottom how
we select the features of the Vision API that we want to use.
Let me also mention the new connectors feature
in public preview for now, which further
simplifies calling some GCP APIs with a dedicated call syntax.
MARTIN OMANDER: Nice.
Very declarative indeed.
But there's still a bit of code, right?
GUILLAUME LAFORGE: Of course.
For example, our initial Cloud Function
was calling APIs programmatically.
But there was a bit of logic and data transformation
code in the middle.
So this logic part stayed but the rest is all declarative
now.
MARTIN OMANDER: Wow, that's less code to worry about.
And also less event handling maybe?
GUILLAUME LAFORGE: True.
Less event handling.
Before with a choreography approach,
we had three different ways to handle events, direct events
from Cloud Functions, Pub/Sub wrapped GCS even
from Cloud Run, and also Cloud Events
payloads with even arc to Cloud Run.
Now it's raised API calls.
MARTIN OMANDER: That sounds awesome.
So what did you learn from this experience?
GUILLAUME LAFORGE: It was super refreshing to call
the microservices via REST instead of different event
handling approaches.
Also, did you see the Workflows console?
MARTIN OMANDER: Yeah, that was the one
with a flowchart and the YAML definition side by side, right?
That seems super handy for understanding the process
visually.
GUILLAUME LAFORGE: You'll get a bird's eye
view of what's going on.
And you can gain confidence in making changes in your workflow
without the fear of breaking a large distributed system.
And here on this screen, seeing all the workflow execution
is nice too.
You know at a glance for each execution
if it succeeded or failed.
I guess it's much easier to debug when things go wrong.
Also, we were able to cut down some code,
even remove some services completely
as they could be called decoratively from the workflow.
Same thing for error handling.
That can be done from the Workflow.
MARTIN OMANDER: OK.
Was it easier to set up and configure?
GUILLAUME LAFORGE: Yes, that was another benefit.
No need to configure Pub/Sub, Scheduler, even Arc.
We only needed to make a REST API call to our microservices.
MARTIN OMANDER: OK.
Any drawbacks you noticed?
GUILLAUME LAFORGE: Yes.
Since it's a different approach, you got to learn it, right?
MARTIN OMANDER: Yeah.
Also, you're actually transforming some code
into YAML configuration.
Does that make it harder to test and debug?
GUILLAUME LAFORGE: Well to some extent, yes.
But on the other hand, it's really just
declarative API calls.
MARTIN OMANDER: OK.
And in terms of architecture of the system, any other impacts?
GUILLAUME LAFORGE: We lost some parallelism along the way
in favor of a more sequential and predictable chain of calls.
We may have lost a bit of flexibility as well.
MARTIN OMANDER: Right.
I guess with event driven systems,
it's easy to extend the system.
You just fire up some new type of event.
Right?
GUILLAUME LAFORGE: Yeah, true.
Or plug into existing even streams to add new services.
MARTIN OMANDER: Ah, right.
But overall, it was a good experience?
GUILLAUME LAFORGE: Definitely.
If you can draw your business processes in a flowchart,
be sure to give Workflows a try.
MARTIN OMANDER: Cool.
And how can we learn more about Workflows
and get our hands on it?
GUILLAUME LAFORGE: Mete and I have
been running a serverless workshop including
both event driven and orchestrated approaches.
You can freely follow the collabs at your own pace.
And the code of our Pic-a-Daily app
is all open source on GitHub.
And you can also read our article with all the details.
MARTIN OMANDER: Awesome.
It's time to read about it and get our hands on the code.
Thank you, Guillaume.
GUILLAUME LAFORGE: Thank you, Martin.
[MUSIC PLAYING]