(gentle music)
- Hey everyone, this is Khushbu.
I'm one of the product managers
working with Azure Container Networking Team.
Today, I have Chase with me,
who is the product manager working with
Azure Kubernetes team,
and Vamsi here, who is working as a software developer
with Azure Container Networking Team.
Today, we'll be talking about topics
like Azure CNI Overlay, Azure CNI powered by Cilium,
Cilium Enterprise on Azure Marketplace,
and an upcoming feature of Network Observability
that our team has been building,
and over to Chase to go more over on the topic,
Azure CNI Overlay.
- Thanks, Khusbhu, appreciate it.
So today, we're going to be talking about Azure CNI Overlay,
but to start that off, let's talk about
what a CNI plugin is real quick.
So a CNI plugin allows Kubernetes
to use container networking interfaces
to communicate between pods and services.
It enables network integration and management,
including IP address management,
and allows for the use of different networking solutions,
such as virtual networks, overlay networks,
and software-defined networks.
The use of a CNI plugin is essential for integrating
and managing networking Kubernetes.
It's important to choose
the right plug-in for your environment
to ensure optimal performance and scalability.
AKS currently offers a few options for CNI plugins,
the first being Azure CNI.
Azure CNI is Azure's plugin for Kubernetes,
and is used for more granular control
over your networking resources.
The next is Kubenet.
It's a basic CNI plugin
that provides simple networking functionality
for Kubernetes clusters.
It's currently the default option for AKS.
And the last option is BYO CNI,
which is bring your own CNI.
This allows you to use other CNI plug-ins not listed here,
and that you already may be familiar with.
Each of these has its own limitations.
Kubenet has a scaling limitation of 400 nodes per cluster,
with 250 pods per node.
And while it's an overlay network,
you can't share your pod subnet across clusters.
Azure CNI has limitations in terms of
the number of pods it can handle,
with a cap of fewer than 65,000 pods.
Since the pods use VNet IPs,
it also has the risk of IP exhaustion.
Next is, BYO CNI is great, however,
it has a lot of overhead management and performance hits
that come with double encapsulation and decapsulation.
In addition to all of these specific limitations,
all these CNI plugin options
require you to create and manage
route tables for your networks.
This can add a lot of complexity
when building and growing your clusters.
So to address these limitations,
we've developed Azure CNI Overlay.
The traditional Azure CNI mode
assigns IP addresses to every pod
from a pre-reserved set of IPs on every node,
or a separate subnet reserved for pods from your VNet.
Now, this can lead to IP exhaustion in your VNet
and difficulties in scaling,
and it can also be difficult to plan.
With Azure CNI Overlay, clusters are deployed
into an Azure VNet subnet,
and pods are assigned IP addresses from a private CIDR
logically separate from the VNet hosting the nodes.
At the time of creation,
each node is assigned a /24 address space
from the pod's private CIDR block.
This isn't limited to the creation time,
and as your cluster scales, each new node that's created
will get another /24 address block
as long as you have space within your private CIDR block.
Also, a separate routing domain is created
within the Azure networking stack
for pods' private CIDR space,
which creates an overlay network
for direct communication between pods,
and for cross cluster communication,
SNAT is used with the node's IP address.
This is all done without extra encapsulation,
that is increased scale up to thousands of nodes.
In addition to this, Azure CNI Overlay
also supports the Cilium dataplane.
Now, to create an Azure CNI Overlay cluster,
it can be as simple as a single CLI command,
by defining overlay of the network's plugin mode
and defining the private CIDR block for your pod's subnet.
You can check out our current documentation
for an update experience to take your Azure CNI cluster
from VNet to Overlay.
Now, this is currently in preview,
but keep an eye out for the generally availability soon.
So let's summarize the key features of Azure CNI Overlay.
You get up to 12 times the scale of the other plugin options
and the flexibility to grow your cluster
to thousands of nodes.
You no longer have to worry about
the management overhead of BYO CNI,
or planning and creating all of your own route tables.
CNI overlay handles all that under the hood for you.
There's also no more worrying about VNet IP exhaustion.
Your VNet IP addresses are for your nodes,
and pods get their own.
With Azure CNI Overlay,
you can share your pod subnet across clusters
for an amazing amount of flexibility in your environment.
With Azure CNI, you also have Windows Server 2022 support,
and you get native host networking performance,
and don't have to worry about the performance hit
from double encapsulation and decapsulation.
Let's just take a look
at a few of the performance metrics here,
and you get a big boost in pod-to-pod subnet
from Azure CNI Overlay
to a typical BYO overlay network solution.
In addition to this, as you can see in the center,
the pod-to-host performance is exactly the same
as our host networking, so there's no more performance hit.
In addition, the pod-to-service
has an extremely high throughput, as well as
when you combine Azure CNI Overlay
with Azure CNI Overlay plus Cilium,
you get an even bigger benefit.
And with that, I'm going to pass it back to Khushbu,
who's going to explain a little bit more
about what that means.
(transition swish)
- Thank you, Chase, for going over Azure CNI overlay.
Today, I'll be talking about Azure CNI with Cilium.
Recently, Microsoft signed a strategic partnership
with Isovalent in which we are now natively integrating
Azure CNI with Cilium into AKS clusters.
This will power network policy and service routing via eBPF
to make it more scalable and performant
than legacy IP tables-based solutions.
This will also enrich clusters with Network Observability,
that will be covered later section by Vamsi.
As you see in this architecture diagram,
Cilium CNI is natively integrated,
and can cohesively work with CNI Overlay
and VNet-based offerings.
What this means is that the pod networking
continues to leverage the Azure CNI solutions,
like Overlay or VNet for provisioning
native routing on the Azure host,
and in-cluster network policy,
and Postman, and service routing
can be programmed via eBPF by Cilium CNI.
So, as shown in Chase's Azure CNI Overlay slides,
our performance runs have seen 14% gains
in pod-to-service routing
between CNI Overlay and CNI Overlay powered by Cilium.
This is powerful, as you are getting
best of both CNI Overlay and Cilium.
However, if you are seeking more advanced features
offered by Cilium Enterprise, then we also have an option
of one click upgrade of Cilium-powered clusters
to Cilium Enterprise image via Azure Marketplace.
As a part of our strategic partnership with Isovalent,
Cilium Enterprise offering
is now available on Azure Marketplace.
You can create new clusters with Cilium Enterprise CNI,
or upgrade existing Cilium-based clusters
to enterprise version.
Post the upgrade, the customers have options and flexibility
to enable more advanced features
provided and supported by Isovalent.
As you can see, again in this architecture slide,
after upgrading the cluster with Cilium Enterprise,
highlighted boxes gets updated to enterprise images
while maintaining the same integration
with Azure CNI Overlay or VNet.
Some of the key benefits of installing
Cilium Enterprise via Marketplace are listed here,
like easy deployment,
customers do not have to self manage enterprise finally,
as they can be auto-updated by new versions
as they get pushed to the marketplace,
and there is zero downtime to the upgrades.
(swoosh transition)
So here, I'm going to show you a quick demo
on how you can enable Cilium Enterprise on your cluster.
So to begin with, as a first step,
you need to create a cluster with the network dataplane
set to Cilium.
I have already set up a cluster,
since it takes lot of time to set one up,
my cluster is ready.
Now that I have the cluster created,
I'm quickly going to check
for the Cilium status on the cluster.
So in the Cilium status, if you look at the image version,
it's the one provided by Microsoft.
But to enable and use the extensive features of Cilium,
you would want to upgrade it to the enterprise version.
And to do that, you first need to go to the portal
and look for Marketplace, and then look for Cilium.
And once you look for Cilium,
you'll see Isovalent Cilium Enterprise.
Just click on Create.
It'll ask you for basic details, like subscription,
your resource group.
And also, you need to select the cluster
on which you want to update the Cilium version
to the enterprise version provided by Isovalent.
Click on Create.
This will start running a process,
where it would upgrade your Cilium Microsoft version image
to the enterprise version image provided by Isovalent.
So once this deployment is completed,
we'll quickly check on the status of the Cilium.
Now that the deployment has been completed,
we're quickly going to check back on the Cilium status.
And when I check for the status, it has been already updated
to the Isovalent version of Cilium,
and now you are ready to use the features that are provided
by Cilium Enterprise version.
(cymbal roll swelling)
So with that, now we have Vamsi
going to take over and talk about Network Observability.
So over to you, Vamsi.
- Thank you, Khushbu, and thank you, Chase.
There is a flurry of updates you guys give.
With that, I want to introduce you guys
to Network Observability.
So this is a new feature, we have been working to cover
some of these lacking functionality in non-Cilium workloads.
As containerized environments
continue to grow in complexity,
it can be challenging to identify the root cause
of networking issues in a given cluster.
So intermittent failures can be particularly frustrating,
and then gaining visibility into your networking stack
can be really daunting for an engineer.
So many organizations are struggling
to find effective solutions to these problems.
So that's where Network Observability add-on
comes in the picture.
So Network Observability helps to gain insight
into networking infrastructure
by collecting realtime data points using eBPF in Linux
and existing components in Windows.
Then the add-on is going to convert these data points
into common set of Prometheus metrics,
which are then scraped by self-managed Prometheus servers
and our Azure-managed Prometheus.
With the help of Network Observability
coupled with Azure-managed Prometheus
and Azure-managed Grafana, you can quickly visualize
all your clusters' network performance
under one single pane of glass with historical context.
So this was not present before.
So it going to help all the DevOps engineers
and engineers who are managing large clusters,
or small clusters for that matter,
like a whole new set of tool set.
So as I was speaking earlier, so this is very easy,
on demand, configurable, easy to enable and disable.
We are leveraging open source, existing open source tools
to build this Network Observability add-on.
With the power of managed Prometheus and Grafana,
we are providing historical context,
and multi-cluster view, et cetera, et cetera.
The good thing is this add-on
currently supports both Linux and Windows.
So here, let me show you what a dashboard would look like,
and then we can go into a demo.
So this is a quick view of dashboard.
So currently, we are supporting node-level metrics,
and pod-level metrics are going to come soon
in the later part of this year,
and then that is going to be even more granular,
even more in-depth view of what is going on in your cluster.
So with that, let me show you a quick demo.
(cymbal roll swelling)
So here, I have a resource group,
and it has an AKS cluster deployed.
This AKS cluster has Azure-managed Prometheus enabled on it.
This Azure-managed Prometheus add-on pods
are scraping the Network Observability data,
and then they are sending the data
to Azure Monitor workspace in native Prometheus format.
And then this managed Grafana dashboard
is now using this Azure Monitor workspace
as the data source to visualize all the data.
So now, if we can quickly open this Grafana dashboard,
which I already did here,
and then here is the dashboard for viewing
all the Network Observability add-on metrics.
So you can see that our data source
is that same demo-managed Prometheus as the data source.
So as I was telling you earlier,
you can see all the clusters
and all the nodes at the moment.
So this is the multi cluster view we were talking about.
Currently, this Network Observability add-on
does metrics at a node level,
which means that all the data points,
everything are aggregated under a node aspect, node labels,
and later on, later part of the year,
we will have pod level, wherein the user can define
like the granularity, like do they want it by service,
do they want it by pod name, et cetera, et cetera.
And then we will see all those are dashboard here too.
But coming back to this node-level metrics,
you can see we have the very basic traffic stats
to quickly see how is it going on this
on an aggregation of all the clusters,
or by cluster, however you want to see it.
Drop stats has been particularly very frustrating
for a lot of engineers, because it is very hard to identify
where the traffic is going, how is it getting dropped.
So within a given VM, the Network Observability add-on
uses eBPF in Linux and existing tools in Windows
to find out, okay, what could be the reason for a drop.
So here, you can see we have some drops of aclrules,
endpoint rules, IP table rule drops, right,
which is particularly problematic.
And then if any other TCP functions
have also dropped some packets, so we get to know.
We are also adding a lot more drop types and drop reasons
as we go further in development process.
So then we move on to the connection stats,
so you can see what is going on
with respect to your remote address counts.
So this is like active connections by remote address,
so you can look through this, or like how are
all the existing TCP connections are by state,
and we can see if there are timeouts, resets,
et cetera,
so this comes in handy when we are trying to see
intermittent failures, to see,
okay, is something like really bad,
or why is my connection getting dropped randomly.
At the bottom, we have the interface stats
to see if everything is fine, and everything is going good,
because with Accelerated Networking support,
there are multiple different metrics we can see
if the actual physical NIC attached to the VM is doing fine.
So yeah, so this is a quick demo
of what is coming with Network Observability.
We will be coming soon with this feature in next month.
(transition swish)
That's the quick demo of Network Observability feature.
Thank you.
With that, Khushbu, take it over.
(transition swish)
- Thank you, Chase and Vamsi, for doing a presentation,
and we are having Network Observability feature
that's going to be coming up soon,
and you're going to see those updates
on our Azure blogs and announcements.
Thank you for watching the video,
and hope you enjoyed the session.
To learn more, you can always go and read more about it
on our Azure documentation.
Thank you.
(hopeful music)