Hey everyone, welcome to today's session, where we are going to focus on how to run your workloads
in Azure Kubernetes Service in a multi-tenant infrastructure.
A common pattern we've seen at FastTrack is customers who have single-tenant, containerized applications
that are looking to run them in a shared Kubernetes environment on AKS.
One of the primary motivations for this is being able to run your workloads in a more cost-efficient manner,
instead of creating bespoke infrastructure for each customer onboarding into your solution,
with the power of the Kubernetes platform, and AKS,
you can now deploy those workloads into a shared environment,
saving on cost, and creating easier to manage deployments.
This does presents some challenges, however.
While containers do provide a level of process isolation,
there are considerations that need to be made
to make sure your runtime environment for customers is secured,
as well as ensuring that you are achieving [a] consistent level of performance for each customer,
and making sure your developers have the appropriate level of access to cluster resources and making sure your developers have the appropriate level of access to cluster resources.
In today's session, you learn about some of the built in features in AKS
and Kubernetes that can help address these concerns and give your customers
the best experience possible on these platforms.
Let's start by defining what we mean when we say an application is "multi-tenant" or "single-tenant."
A multi-tenant application is an architecture in which a single instance of your application
is serving multiple customers, and each customer is considered a tenant.
This can be running on any underlying platform, whether that's a single server in a data center,
or a Kubernetes cluster running in the cloud.
When you deploy your application, you're only deploying it once, regardless of the number of tenants you're serving.
A single-tenant application, on the other hand, is an architecture in which a single instance of the application,
and the supporting infrastructure, serve a single customer, or "tenant."
So, what do we mean by "multi-tenancy" in a Kubernetes cluster, then?
In Kubernetes terms, multi-tenancy is concerned with providing isolation,
and fair resource sharing, between multiple users and workloads running within the cluster.
These workloads can be instances of the same application,
or, many different types of applications.
Now, let's take a look at the different types of isolation models for AKS.
Kubernetes environments, in AKS or elsewhere, aren't completely safe for hostile, multi-tenant usage.
Kubernetes and AKS have tools to make exploits more difficult,
but in certain scenarios, physical isolation at the hypervisor level is the only level of security you can trust.
With this type of AKS architecture, nodes will have a lower pod density, and the cluster will have more idle compute.
This will lead to more expensive deployments, and more management overhead, as he now must maintain multiple clusters per customer.
This is only recommended for your customers who may be in highly-regulated environments that require the absolute highest level of isolation.
Kubernetes provides features that let you logically isolate workloads in the same cluster.
A namespace in Kubernetes creates a logical isolation boundary similar to the concept of resource groups in Azure.
With this type of AKS architecture, nodes will have a higher pod density, because
they are running more workloads, and will have less idle capacity.
This, combined with auto-scaling nodes, will maximize the cost savings.
Additionally, this will reduce management overhead, as there will be less clusters to manage.
This isolation model is considered the best-practice approach for AKS in most customer scenarios.
In the SaaS provider model, each consumer of your service would be considered a tenant,
and would have its own namespace dedicated to where you would deploy the components that make up your application.
Namespaces by themselves do NOT provide a level of isolation between the different workloads.
Kubernetes defines a set of building blocks, or "primitives", which collectively provide
mechanisms that deploy, maintain, and scale applications.
The ones we are focused on today are some of the ones that going to help achieve
and implement a successful multi-tenant infrastructure.
These building blocks are divided into three main categories:
Access control - who can do what in the cluster...
... that includes Role-Based Access Controls and Identity and Access Management,
Resource sharing - ensuring consistent and fair performance for tenants - that include[s]: quotas, priority priority, limit ranges, pod affinity,
and node affinity,
and, runtime isolation
enforcing a secure runtime environment,
including network policies and pod security policies.
We'll take a look at these building blocks as well as look as some of the
AKS-specific implementations that you can take advantage of.
"Role-based access control" is a method of regulating who can do what operations to which API resources within your cluster.
The RBAC API in Kubernetes declares four kinds of Kubernetes object[s]:
Role, cluster role, role binding, and cluster role binding.
A role always sets permissions within a particular namespace. When you create a role, you must specify the namespace it belongs in.
Cluster role, by contrast, is a non-namespace resource.
A Kubernetes object always must be either namespaced, or not.
AKS allows you to enable Kubernetes RBAC at cluster creation time.
To interact with your AKS cluster, you must first pull down your kubecfg file, which contains the cluster connection information.
The Azure CLI provides an easy command to pull down that file.
You can use Azure RBAC to control access to these credentials.
These let you define who can retrieve the kubecfg file,
and what permissions they have within the cluster: "admin" or "user".
For enhanced security,
you can enable Kubernetes RBAC, and integrate with Azure AD for user authentication
to limit access based on the user identity or group membership
within your Azure AD tenant.
The two panes on the lower right of the screen show an example of utilizing Kubernetes RBAC with Azure AD integration.
We have created a role here that is namespaced to "dev" that gives full permission to that namespace.
Then, the role binding assigns that role to the group object ID
of the Azure AD user, group, or service principal you want to get this level of access to.
Alternatively, you can enable Azure RBAC for Kubernetes authorization,
to avoid having to manage Azure RBAC and Kubernetes RBAC separately.
This allows you to use the access control blade of the cluster resource or through the Azure CLI
to scope them to a specific namespace. To assign roles to users or groups,
AKS provides four built-in roles today, and you can also create custom role definitions.
The pane in the top-right shows an example of assigning a role at a namespace level.
When you have multiple customers running in a shared cluster,
you want to avoid having any one customer or tenant hog cluster resources, and cause performance issues for the other tenants.
To address this concern there are a couple of options in AKS.
First, you can deploy resource quotas to limit total memory, CPU, and storage that pods can consume,
as well as how many objects of each type - pods, load balancers, ConfigMaps, etc. - on a per-namespace basis.
Resource quotas are independent of the cluster capacity;
they are expressed in absolute units.
So, if you add nodes to your cluster
Or, your cluster node scales up,
this will not automatically give each namespace the ability to consume more resources.
The other option in AKS is to use dedicated node pools for a customer.
The Kubernetes scheduler can restrict what workloads can run on which nodes using taints and tolerations.
A taint is applied to a node that indicates only specific pods can be scheduled on them.
A toleration is then applied to a pod that allows them to tolerate a node's taint.
As an example, assume you have a node in your AKS cluster for nodes with GPU support.
You define a name - such as "GPU" - then a value for scheduling.
If you set this value to "NoSchedule", the Kubernetes scheduler can't schedule pods
on the node, if the pod doesn't define the appropriate toleration.
With a taint applied to the nodes, you can then define a toleration in the pod
specification that allows scheduling on the nodes.
This example defines the "SKU" "GPU" and the effect "no scheduler" to tolerate the taint applied to the node.
This approach will let you run your customer's workload on dedicated nodes, and as those nodes scale up,
your customer's deployment will be able to take advantage of the additional resources.
If you have critical pieces of your application that you want to ensure get scheduled before less critical components,
you can use pod priority.
you can use pod priority. You assign pods to a PriorityClass, which is a non-namespaced object that defines a mapping from a name to an integer value
of the priority - the higher the value, the higher the priority.
Pod priority indicates the importance of a pod relative to other pods, and the Kubernetes
scheduler queues the pods based on that priority.
Pod preemption allows the cluster to evict, or preempt, lower priority pods, so that higher priority pods can be scheduled if there's no
available space on a suitable node within the cluster.
You can also combined resource quotas with pod priority classes, to ensure that high priority
or low priority pods get the appropriate resource budgets.
While resource quotas can restrict the available resources in each namespace, that will not prevent a pod or container
from consuming all the resources within that quota.
By default, containers run with unbounded compute limits. To prevent a pod or container from monopolizing all available resources,
you can use a LimitRange policy on the namespace.
For example, let's say you set a resource quota for the Dev namespace at 8 GB of RAM and 16 cores.
You could constrain pods in the Dev namespace to request a minimum of 200 millicpu and a max limit of 500 millicpu
and a default limit of 500 millicpu.
Now, whenever a container is created in the Dev namespace, Kubernetes will perform these steps:
If the container does not specify its own CPU request and limit, it assigns the default CPU request and limit to the container.
It will verify that the container specifies a CPU request that is greater than or equal to 200 millicpu,
and it will verify that the container specifies a CPU limit that is less than or equal to
500 millicpu.
The maximum and minimum CPU constraints imposed on a namespace by a Limit
Range are enforced only when the pod is created or updated.
If you change the Limit Range, it does not affect pods that were created previously.
This is a critical policy to enforce so that you can ensure that you don't deploy a workload into your cluster
that will cause performance issues for your customers.
The pod affinity and anti-affinity is a scheduler feature
and a great way to control how pods are distributed to nodes in your cluster based off the labels on pods that are already running.
For instance, you might want to constrain a pod
for a particular tenant to try to run on the same node for performance reasons.
Or, you may want to ensure that one of your services is distributed across as many nodes as possible, like in this example above, for
a Redis deployment, that ensures that instances of Redis will be distributed across each node
in a three node cluster.
By default, pods in Kubernetes and AKS are non-isolated;
they will accept traffic from any source.
In order to control the ingress and egress traffic flow, a Network Policy will need to be applied.
Network Policy is a Kubernetes specification that defines access policies for communication
between pods, similar to the concept of Network Security Groups in Azure.
Using Network Policies, you define an ordered set of rules to send and receive traffic, and apply them
to a collection of pods that match one or more label selectors.
You choose a network policy option when you create your AKS cluster,
and the policy option can't be changed after the cluster is created.
AKS provides two ways to implement network policy -
Azure's native implementation, called "Azure Network Policies",
or Calico Network Policies, an open source network and network security solution.
Both implementations use Linux iptables to enforce
specified policies and are thus only available for Linux-based nodes at this time.
To enforce a secure runtime environment, and protect against exploits from impacting other
tenants within a cluster, you can use Pod Security Policy.
A Pod Security Policy is a cluster-level resource that controls security-sensitive aspects of the pod specification.
They allow a cluster administrator to control things like the running of privileged containers,
the SELinux context of the container,
the AppArmor profile used by containers,
and the usage of filesystem volumes and host networking ports.
In an AKS cluster, you can integrate with Azure Policy to enforce security standards.
An admission controller is used to intercept requests to the API server when a resource is to be created or updated.
The admission controller can then validate the resource requests against a set of rules on whether or not it should be created.
An AKS cluster with this integration enabled
can use built-in Azure Policies which secure pods, and other Kubernetes resources, similar to Pod Security Policy.
The Azure Policy add-on for AKS installs a managed instance of Gatekeeper, which is a
validating admission controller, and is built on the open source Open Policy Agent.
These policies can only be applied to Linux node pools,
and only the built-in policy definitions from Azure are supported at this time.
These policies can be scoped to your subscription or management group,
so that you can enforce the security policies in a consistent manner
across your Azure environment.
You can even apply these to any Kubernetes cluster you have enrolled in Azure Arc to secure
your Kubernetes environment, no matter where it is running.
The topics covered here will help ensure your customers have a good experience on the AKS platform.
As these are evolving platforms, make sure to check out our docs and the Kubernetes
docs to get more information and details on what we have covered.
There are many tools available for Kubernetes that are outside the scope of this session that can
augment or add additional functionality to secure and control the runtime environment,
but the ones outlined here today
are available to you by default, so make sure to take advantage.
Thank you!