>> Hi, I am Donovan with another episode of Azure Friday.
I'm here with Brendan, and we're going to learn about
this research project called
Metaparticle. Welcome to the show, Brendan.
>> Thanks for having me.
>> So, tell me real quick,
what do you do here at Microsoft?
>> So, I'm in charge of a bunch of
different teams that I would say
work on DevOps and Azure.
>> Okay.
>> So, from containers to
more traditional things like
the Azure Resource Manager templates,
but really focusing on how
people build and deploy their code into Azure.
>> All right. Great. And this Metaparticle,
tell me what this is about.
>> So, Metaparticle came about
really because I think a lot
about the complexity of going from code to cloud.
There's way too much in the way of tools to learn,
and different configuration languages,
YAML, all this sort of thing that
stands in between you with
some code on your computer
and a reliable system running on the cloud.
And so, I think, really,
we need to, as an industry,
think about how do we build new tools
to reduce that complexity and
make it more approachable for people?
And Metaparticle is maybe one step down that direction.
It's a long way to go, I think.
>> All right. So, is this supposed to be replacing
my traditional CICD system or is
this something that's going to
augment that to make it easier?
>> I view it more like the inventions of
things like object-oriented programming
or managed code, right?
It's like, in a more traditional language,
you might have to worry about memory management.
In a managed code environment,
you've got a garbage collector.
It's just trying to simplify
things and really focusing on the developer's experience,
not necessarily on something like
CICD that takes you in a production roll-out.
>> Got you.
So, we're talking about more of that inner loop,
that developer loop that's happening on my dev machine.
This is going to make that part of my life easier.
>> And, I think also bringing somebody
from writing a computer program that runs on one machine,
to writing a program that runs on the cloud.
I think that's a big hurdle for some people
and that's really what it's focused on,
is trying to break down those barriers.
>> Awesome. So, let me see how it works.
>> Sure. So, I'm going to start out here
with showing a really simple JavaScript Node.js server.
It just says, ''Hello World,'' and it's going to
serve on my local host right now.
And so, relatively easy for me to run it.
It's there up and running,
you see, and they can see the requests down below.
There was one that failed when it started to come up.
And this is what I would expect a developer,
you know, is ready to do.
They understand how to write a little bit of code,
maybe take a sample off the web,
run it on their local machine and they
can access it via HTDP.
>> Sure.
>> But then, you go, ''Well, okay,
I want to take that and I want to
package it up and run it on Azure."
Right? And if I want to do it with
containers and I want to do it with Kubernetes,
there's suddenly a bunch of tools
that you need to learn, right?
You need to learn how do I use Docker?
How do I create a Docker file?
How do I use Kubernetes?
How do I turn up
and package my application and deploy it?
There's a bunch of different steps and I think
none of them are necessarily really related to your goal,
which was, how do I get my code running in the cloud?
And so, let's take a look at what we do with Metaparticle.
So, with Metaparticle instead,
here's that same code
but we've augmented it a little bit.
One thing you see here is I've pulled in a new library.
I've pulled in this Metaparticle library.
It's important to point out that Metaparticle is
actually a polyglot solution.
I'm showing it to you here in Node.js.
In a little bit I'll show it to you in Java but
we've got implementations for Python,
for.NET, for Go, for Rust.
The community has started to spring up around this to
provide a bunch of
different implementations in different languages.
>> Okay.
>> All meant to kind of feel
natural inside of that programming language.
And so, you see the library that I've pulled in here,
and then you see this containerized directive.
And so, this containerized directive has said, ''Okay,
I want you to actually take that code that I was running,
which is down here,
and package it up as a Docker container,
publish it to a Docker registry.''
In this particular case, it's going to run
locally on the machine using Docker,
but in later examples, we'll show how you can also
then push the code out to a Kubernetes cluster.
And so, let's take a look at that.
So now, up here, you can see the Docker build happening
and actually it's also pushed it up to the registry,
and that was actually the same
experience that I had before.
I see the logs from the application up and running,
but now, if I take a look,
I've got a Docker container up
and running. So, how did I-?
>> So, I missed where you actually executed.
So, do I just do a normal node command?
>> Correct. You just say,
exact same file, you say, Node server.js,
but the library itself says,
''Ah, you know, I see that you are running this program.
I see the request to containerize it.
I'm going to package it out of his container,
push it to the registry,
and then when that container runs,
it actually detects that it's running inside a container,
skips that whole part and then just runs your code."
>> Got you. Okay.
>> Right. So, it's this mixed mode where
if you run it on your laptop, it packages itself.
If you run it inside a container,
it runs your application code.
>> Awesome.
>> And I think that really helps
make the experience feel the same, right?
Because when you run it, you still get standard out,
you still get standard in,
and you still get that, sort of,
like, it feels like it's running on
your machine but it's
actually being packaged and run inside a container.
>> Awesome.
>> I'ts really trying to preserve that, like small steps.
We want to go from running on your laptop,
to running in a container on your laptop,
to running in a container on the cloud,
and really how to look and feel the same way.
>> Perfect. So, one thing I didn't see you
do is you didn't, where's the Docker file?
Because it's almost like I know too much about it.
It is a lot of black magic-
>> No, but I think that's really important actually,
and actually what it does is,
it does write the Docker file to your file system.
And so, if you learn about that at
some point or you need to tweak it in some way,
or you're done with
this library and you want to just move on,
that Docker file is still there.
It's something you can build from,
and I think that's critically important with
these systems where you have to be able to,
where I say it's more like
you decide to learn how to mow your own grass.
Well, in the first week, maybe don't do a good job.
Grass gets a little bit longer, but
it's not a catastrophe.
I think a lot of these magical systems,
it's like jumping out of a plane with no parachute.
It's a bad way to learn.
>> Sure.
>> You want to learn in a sort of easy approach.
And so that's what we're really intending to do.
>> All right.
>> All of the pieces are
there if you start looking for them,
but the experience at
the top level is also compelling and easy to use.
>> Awesome.
>> All right. So, and then you can see here,
we're hitting that server just like we were before.
>> Correct.
>> All right. And so,
I'm going to go over to
a slightly more complicated demo this time.
This time is in Java,
and I'll give you a sense for
how we're using language level features.
And so-
>> Do you use Maven to build this-?
>> Yes, we are using Maven to build it now,
and it's building up a directory.
And in the case of Java,
the annotations are actually
traditional, like app annotations.
>> Okay.
>> So, instead of having some of
it feels like Node, it feels like Java.
>> Sure.
>> If you're a Java programmer,
those annotations are traditional and familiar to you,
and you can see, but still the Docker build is happening.
It's actually pushing it up, and now,
but what you'll notice here is
in that Java directive that I had,
I actually asked for four replicas,
and I asked for it to be deployed to Kubernetes.
And so now, instead of just running
it in a single container,
I've actually deployed four versions of
the same container up into Kubernetes.
These are those replicas that you see right there.
Web one, web two, web three.
These are the hosts, these are the various containers.
And in fact, if I run kubectrl now,
I can see that I've created
a load balance service, my web service here.
And I've created four replicas of that service.
So my code is no longer one instance of my code.
>> Right, so those.
>> So there are four different replicas of my code.
>> Yeah, those annotations would have been
actually in my Java classes.
>> Exactly.
>> Not in my Palm File.
So was there a different,
when I ran Maven,
did I just do the normal package?
How did it know to go even read
those annotations before you fire
up my war file to do this?
>> It's actually the execution of the JAR file, right?
So it is compiled into the,
just like with the no Gess example,
when you say Java JAR,
and you run that Java application on your local machine,
it sees those annotations.
>> Sure.
>> And it packages until.
>> I guess I missed that step.
I figured you were doing a Maven build.
>> But the Maven build happens inside that step. Sorry.
>> Okay, so I missed the step there.
>> So obviously, when you run that program,
if you've made changes to your code,
we need to detect
those changes and build them into the container.
>> Okay.
>> So part of building the container
is actually running a Maven build.
>> I see.
>> So in order to pick up those changes,
you have to have done
the build inside the context of the container.
Otherwise, they wouldn't be there, right.
Otherwise, you'd have a container that
was stale effectively.
>> Got you. So it was when
I actually executed my JAR file,
that the Maven kicked off to rebuild my image.
>> Rebuild your image. Run the docker
build to flood Kuban.
>> Right. So I would have done a Maven build even
before that to build the JAR based on my chain.
>> That's right.
>> That's the part that was missing.
So you get a little bit more like,
I think we jumped a couple of steps there.
I'm a developer, I made a change,
I'd set the annotations, I ran Maven,
I got my war, my JAR, whatever it was.
I want to execute that and then that's when
the second Maven kicked off to go build my path.
>> I mean the way I usually do it,
I just use Maven exact
Java and it'll just do all that for you.
>> Got it. Fair enough.
>> You don't have to do those two steps.
>> Okay. Got it.
>> Either way.
>> Perfect.
>> So now, if you look, we
have the replicated deployment.
And now, I can actually jumped into
my cluster just to show
you that it's actually up there and working.
And I get a little shell up inside my cluster.
And I can do a little for Lukins so for.
So there's two interesting things to note here.
One is that, you can
see that it's moving through a bunch of different hosts.
So it's really hitting that load balancer,
it's really going between these different replicas.
But also interestingly up here in my standard out,
I'm getting all of those requests.
So on my machine,
it feels like the process is running on my machine.
Standard out is still kicking stuff to my standard out.
But actually, it's running up in
the cloud in an Azure container service cluster,
replicated out of Kubernetes, right.
So I think again, you're trying to keep
that fidelity of developer experience.
>> Okay.
>> So and then of course also, when I leave.
>>And these I'm just assuming
when you went into Kubernetes Azure,
you're using ACR, right?
>> Yeah, you can push.
I mean, you could honestly push it to
whatever registry you want, and then officially.
>>The annotations also, how do I then
off against my registry? Just curious.
>> Right now, actually,
you have to have pre-authorized Docker.
So you have to have run Docker log in already.
>> Okay. Got it.
>> Okay. Little bit maybe
I should automate that as well but.
>> I'm just curious because I was concerned
that there would be credentials in
my annotations in my code that I'm going to commit.
>> No it's done. It's expected
there's a side process somewhere else.
>> Fair enough.
>> You don't need to log in and locker knows about it.
>> Understood.
>> Okay.
>> And actually you can see here,
I hit Control C on that Java process.
And actually, it's tearing down
this condition Kubernetes, right?
>> Sure. Okay.
>> So the idea that it
really feels like it's running as a program on
your machine holds even to Control C in
that program and having the program destroy itself.
>> So all that the control
that I would have to
normally do has all been done for you.
>> All been done for you.
>> Exactly. But it didn't happen or anything.
>> Exactly.
>> I love not having to learn new stuff.
>> And so, then I want to show you this last example.
>> Sure.
>> And I'll give you a little bit of a flavor for
those Java annotations that we're talking about.
>> Okay.
>> So here you can see.
So these are actually the annotations.
I have this runtime annotation here that describes it.
And I have this package annotation that sort of tells you
what repository and what's
the JAR file I want you to package up.
>> Okay.
>> This one's a little bit different actually,
and this is where we start to get into
the cloud native stuff a little bit.
In this case, we're actually going to do sharding.
So in sharding, instead of
round robin and your requests everywhere,
a particular request goes to
a particular machine all the time.
>> Okay.
>> Different requests will go to
different machines but the same request,
and that allows you to do better caching,
it allows you to have sort of better failure modes.
It's a relatively straightforward concept for people
to understand I think but
it's a hard concept potentially to implement.
>> Okay.
>> Right. And so Metaparticle here is
trying to help you with that implementation.
And in particular, what you're seeing here
is I want there to be four shards,
and I want the sharding pattern to be this, right.
So the thing that I want to use is
my sharding key is defined by this regular expression.
Okay. Which in this case is the user.
>> User, yeah.
>> And then, we give
the same containerized directive here.
And then, everything below that is
just a really simple Java web server.
>> Interesting.
>> And so, we'll do the same thing.
We're going to kick off that
the Maven execution of my class.
It's going to kick off an internal build,
package itself up as
a docker file so the build
should finish in just a second.
>> You'll see that to registry.
>> Yeah. So you see the docker build is starting now,
and now the registry push is happening.
I have to admit, I've pushed this image before and
so only the last layer of need should
be pushed in. So it's pretty quick.
And now, this point it's deploying it's up to Kubernetes.
And this time, it's actually going to deploy itself.
It takes a little bit more time because it's actually
deploying those shards and it's actually
waiting for each shard to go healthy.
Kubernetes waits for each shard to go
healthy before moving onto the next one.
>> Okay.
>> Just saying give me four all at ones.
>> Right.
>> But at this point,
all four of those shards were set up.
And what you can see here is
now instead of having one load balancer,
I actually have to load balancers.
One that's out at the edge.
A first layer of
shard router which is the thing that takes
all the requests and decides
which shard to direct traffic to.
>> Okay.
>> And then a load balancer
is in front of my application.
>> Right.
>> And so, it's what's important here is now,
I'm not just deploying your code my code.
I'm actually deploying some utility classes
as well that know how to do this sharding.
>> Gotcha.
>> So now, you're sort of starting to view containers as
libraries that you can kind
of link into your application.
These generic shard router,
I don't know where it came from.
But I'm using it and it works right.
And so I think that's also part of the power is
that it allows us to share our code together.
>> Okay.
>> All right. So there's the deployment.
There's my code up and running.
And now, and you can see all the pods are now,
all these different containers
up and running in my cluster.
Now when I jump into that cluster and do the same thing,
command prompt, and say, wget.
>> I see you here on this,
you show us the sharding actually happening.
>> Right. And so now, you can see here,
if there was load balancing,
I would expect my subsequent request.
>> That come from someone else.
>> Someone else but you can see here,
as they are continually going from the same request,
and if I go in and I change this to donovan,
then it's coming from a different request. All right.
>> Awesome.
>> And so, I've actually,
you can have sort of a rough notion
of this is what sharding means.
Know at right one regular expression and take
advantage of this and start cloud
native pattern in your existing code,
in your existing server.
And so I think, that kind of shows
via this annotation that feels natural.
>> Yes.
>> Just like you are a Java programmer
feels like your dot net program.
>> Right.
>> Whatever else. So hopefully,
that gives you a picture for sort
of where we're trying to go with Metaparticle.
There's a site that describes this has tutorials,
a bunch of tutorials for
all the different languages we support.
And definitely encourage you to go check it out.
And also it's up on GitHub,
and we really trying to build
a community around it because as I said,
it's an experiment, research Project.
But I think it's an important part of
building a cloud native set of development.
>> Perfect. So you're accepting
pull requests for other languages and things like that.
>> Absolutely. In fact actually, both the Python
and Golang came from
community members. That's been really good.
>> Fantastic. That's fantastic.
So we are learning all about
Metaparticle here on Azure Friday.