>> We're about to have a really intense discussion.
This is about how architecture is changing.
Specifically, how does machine learning
change what we think about when we think
about architecture. Join us.
[MUSIC]
>> Welcome to the Azure Enablement Show,
where we'll be discussing the challenges
you and other text-heavy customers have encountered.
Together, we'll be talking with experts to find out
how they think about these problems,
recommended tools, and best practices,
and tips they've learned from
years of experience that you can use.
Today on the Azure Enablement Show we do
something we've never done before.
We're going to take our script and we're going to
simply rip it up,
because we want to talk today about
architecture and architecture is discussion.
Hey, so we're back with Uli and Eric,
talking about one last context
about ways that architecture is changing.
The one that I was hoping we talk about is stuff around
machine learning and what I've heard
called Software Architecture 2.0.
But I think we should do what we like to do,
which is level set and get just a little bit
of context of what that means.
Uli, can you say a little bit about what we're
talking about in this episode.
>> Yeah, perfect. First of all,
let's define what machine learning models are.
At the end of the day, there are techniques
to encode statistics or mathematical models,
and utilize data to effectively evaluate decisions.
That's really in a nutshell what it is when you boil it down.
The transition from software 1.0,
which is imperative code,
somebody typed in if,
and else statements, and so forth.
To software 2.0,
effectively says that parts or all of
the application is being replaced from imperative code,
C#, Java, Python,
whatever it is, to these models.
Actually, funnily enough, the models are built in Python mostly.
The idea is that you take fixed functionality,
because once you write a piece of code,
it doesn't change its behavior,
and replace it with a model that
evaluates based upon the data it sees,
the data it gets,
and data it uses, and evaluate decisions.
I have a great example.
One of the environments that we're using for collaboration a
lot actually replaced 150,000 lines of C++ code,
which was effectively a set of policies that determined,
if I connect to a back-end,
what is the jitter or what's the network look like?
I have various network connections I could pick,
and based upon this very fixed set
of code, I will make a decision.
Now the problem is that this jitter and everything else
continues to change with 5G and whatever else you might do,
and we always had to update the code to do new things.
With the model,
it actually is the data that informs the decision,
and it really is a fundamental change.
When you think about machine learning AI,
today most of this is in the context outside of an application.
I do risk analysis,
or fraud detection,
those things, but there are services that I call and bring in.
This example that I used with
this collaboration environment is not part of the application.
You have imperative code that does certain things,
and you have models that do certain things,
and they're packaged as one.
That I think is the real meaning of software 2.0,
where you replace on the one side imperative code with models,
and on the other side,
you bring the models closer into
the application context and make it as one thing,
which then has its own challenges that we talked about before,
and Eric has some great ideas
in terms of what you need to think about here.
>> Yeah, the way I interpret what Uli said,
and the way I break it down for myself is that,
there's deterministic code and there's probabilistic code.
Deterministic code, it does the same thing on day one,
if you don't change, it'll do the same thing on day 365.
Models, however, operate in realms of probabilities,
which means the way that a model is trained based on a dataset,
it may operate at
86 percent accuracy on day one, and if you leave it,
like locking a puppy in a closet,
it may deteriorate over time
and become super, super low confidence.
Maybe a little dark there, but that's
the essence of what you're doing.
You can't lock these things in
closets and assume that they're going to operate.
There are a couple of things that we need
to look out for as architects.
When we do these things that Uli mentioned,
which is to inculcate models or bring them
into the application and
put them into critical path of functionality,
we have to continually be able to monitor and watch them.
Not from an input,
output, I'm calling this module,
I'm calling this service,
and is there a dial tone,
or did it reply back, and I'm good.
It's more of, what did it tell me,
and is what it told me confident?
Do I have a high degree of confidence
to the quality of what it told me.
If I don't, it's equivalent to an application down situation,
because I could be denying people loans,
or airplane seats or tickets,
or some other crucial element of functionality.
I think that those are the things that application architects,
we need to worry about.
They're not just microservice
architectural components to be included,
and if they're responding, everything's good.
We need to look at the quality of what they're telling us.
>> Again, I would add two specific examples on this one.
One is called Model Drift,
where the model is drifting based
on the data that gets fed into the wrong realm.
Do they have a virus scanner for files?
We need a virus scanner for data,
where the data that it's getting fed,
if it's maliciously done,
then obviously the AI model
might do malicious or stupid things
based upon the data it get fed.
If you can control the data, that's great.
If you get input from random sources,
you need to make sure that the data is accurate,
doesn't do bad things and so forth.
There was a very famous example where
Microsoft put up a bot that was very,
very nice from a functionality perspective,
but people figured out how to make it do hateful things,
because they were feeding it the wrong data.
Part of this is Model Drift,
really deeply understanding the data that's being
used to feed and drive the model,
and new considerations that come up with software tool.
>> What it's interesting to me
is there's echoes from our first discussion together,
in which we were talking about what
reliability or resilience means.
Reliability is not just availability or latency,
but it's also correctness to a certain thing.
Also where, Uli,
when you talk about a virus scanner,
it's not always malevolent things,
it can also be absence.
We know very clearly that there's
some real problems around models that are fed incomplete data that
are including a really good sample set from different races,
classes, or stuff just for humans.
It's very possible to inadvertently skew the data by
not feeding it
a really good representation of what you care about,
and as a result, stuff goes on. It looks like this.
>> Actually, this topic is
becoming big because my brain is going all over.
There are entire fields of study from adversarial machine
learning to our own work in responsible AI,
we should probably discuss.
Might make for a great future topic for us to
dive in on and talk to
people about how we do these types of things.
>> Okay. Let me bring it back to
architecture and then we'll cut this down,
because it seems like we can go differently.
It seems to me like one of the key things
that is a change when we're thinking about
software architecture 2.0 is
our relationship and our understanding of the input.
There's two flavors of input,
there's the input to the model,
and there's the input coming in to be evaluated by the model.
But that for sure is
one way that architecture is changing around this.
Is our cognizance of that,
we have to turn that dial way up.
Are there any other architectural things that
happen as a result of this situation,
that we can say advises architects,
like pay attention to this
or here's a big change that you should be aware of.
>> Eric mentioned it and that's the big one,
which is, we're moving from deterministic behavior,
it does the same thing every time,
to dynamic behavior, which is relative.
It tells you, here's what I think it is,
and here's my probability of what I believe is right.
Therefore, the person that
invokes the model and gets the results back,
needs to determine what do I do with this result,
because it's not deterministic.
That is certainly a great opportunity,
because the system does behave dynamically,
but it's also a burden.
Part of what you need to think about is,
if you have a safety concern system,
like for example, autonomous driving,
can you use models in the driving decisions?
Most likely not, but you can
use the models for perception management.
When you're reading the camera and understanding what's there,
the system will give you probability that this is
a concrete wall and then you need to make decision,
"Well, if it's a concrete wall with
this higher probability, I will stop."
But that has to happen every time.
You need to really think through how you
use deterministic code and these models,
and compose them together for best effect.
>> As a last point for me,
in addition to what Uli said,
don't underestimate the technical architecture
and process architecture required
to keep these models up to date,
to keep them trained, and to keep them
deployed both in the Cloud and at the edge.
It can't be an afterthought.
Start planning that when you start,
as soon as somebody says, "Hey,
this is really in the realm of a model."
There's a whole architecture around MLOps,
we call it here at Microsoft,
about doing all of
the lifecycle management and maintenance around training,
redeployment of these models.
That's super important to think about.
>> That's a fabulous place for us to end up this conversation.
I know we have more topics to talk about,
I'm hoping you'll join me for some of those too.
>> Absolutely, we love to.
>> You got it.
>> That was a really cool set of conversations.
I hope you get a chance to watch those episodes.
We're going to move on to another topic
here on armchair architects,
as part of the Azure Enablement Show.
[MUSIC]