Kustomize is an increasingly popular tool for generating Kubernetes manifests, and is now included with Kubectl 1.14. Rather than using templates Kustomize works by applying modifications to already valid manifests. Using this pattern, it provides various features including resource namespacing, modification of metadata, and generation of Kubernetes Secrets.
To start using Kustomize you need one or more Kubernetes manifests and a
kustomization.yaml file. The
kustomization.yaml file is itself a manifest, which specifies a list of resources, patches to apply, and various other options. The docs give this example file.
Manifests that give complete Kubertnetes resources are listed in the
kustomization.yaml as resources. Manifests can also be used as patches, where they are strategically merged with existing resources with matching metadata.
To modify a kustomization further, or create variants of it, Kustomize uses overlays. Overlays also contain a
kustomization.yaml file, and can include manifests, either as new additional resources, or to patch existing resources. The overlay’s
kustomization.yaml must specify one or more bases. The bases are directories containing
kustomization.yaml files, and optionally manifests. The overlay can then build on these bases, and is also dependent on them. Overlays can also specify other overlays as bases.
All of the Kustomize files can be found in a GitHub repo so you can try the demo out yourself and dig into what Kustomize can do. Please report any issues you find on the repo! The demo repo contains two directories,
overlays/, as well as a
Before running the demo make sure you have Kubectl 1.14 installed.
Alternatively you can use an older Kubectl version and install Kustomize separately.
Additionally you will need access to a Kubernetes cluster with an ingress controller to deploy the app. As well as a domain (or subdomain) that you can point at your Helloweb deployment to use cert-manager.
The initial Helloweb manifests can be found in the
base/ directory. They will work as they are, however a simple
kustomization.yaml is used to add a label and Namespace to the resources:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization commonLabels: app: helloweb namespace: helloweb resources: - namespace.yaml - deployment.yaml - service.yaml - ingress.yaml
Using the standalone Kustomize command the modified manifests can be generated and printed to the terminal with:
kustomize build base
With Kubectl 1.14 the equivalent command would be:
kubectl kustomize base
The manifests can be applied, either using separate
kustomize build base | kubectl apply -f -
Or using the new Kubectl 1.14 integrated Kustomize functionality:
kubectl apply -k base
Thought the rest of this guide the new method will be used, though either should work and have the same effect.
This will deploy the minimal resources to get the app up and running in a Namespace called
helloweb. Wait for the Ingress to be allocated a public IP address and then enter this into a browser to confirm the app is working. It may take some time, even after the IP address is allocated, for you to be able to load the app. When it’s working it should display the message
Hello, world!, with the page using plain HTTP.
Point a domain at the Ingress' IP address. Wait for the DNS record to propegate and confirm that you can still access the page at the domain you have used.
The first overlay builds on the resources created by the
base. It adds a cert-manager Issuer and Certificate, as well as modifying the Ingress to point it at the TLS Secret that cert-manager will create. This shows how Kustomize can be used to enhance sets of manifests. It also makes it easy to keep the base manifests separate from the ones responsible for adding cert-manager.
The cert-manager overlays
kustomization.yaml looks like this:
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base commonLabels: app: helloweb namespace: helloweb resources: - issuer.yaml - certificate.yaml patchesStrategicMerge: - ingress.yaml configurations: - cert-manager-configuration.yaml
This starts by specifying the location of the
base directory which contains the basic Helloweb deployment.
Note that the base specification is the only one that can be ‘above’ the current directory. Kustomize allows for sub directories and does not enforce any specific structure, but it does not allow resources to be used from directories ‘up’ from it. This is enforced for for security reasons, for example to prevent a
kustomization.yaml from pulling private information from elsewhere on a users filesystem.
The labels and Namespace are the same as in the
base/kustomization.yaml, these are applied to the cert-manager resources. The overlay doesn’t inherit the labels and namespace defined in the base layer, but it is possible to override the base layer from within the overlay layer.
resources specification is only needed for new manifests, everything specified in the base
resources is also used in the overlay. This adds a cert-manager Issuer which uses Let’s Encrypt Staging, and a Certificate which uses this Issuer.
patchesStrategicMerge allows partial YAML files to be provided, which are then patched on top of resources of the same group, version, kind and name. In this example it is used to modify the Ingress, adding the TLS specification with the Secret that will be created by cert-manager.
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: ingress spec: tls: - hosts: # Change this to your own domain - demo.example.net secretName: secret-demo
The top section, giving the
name is needed to match the patch to the resource it should be applied to. New items can then be specified in an additive way; so the
spec: definition will match the existing definition of the same name, but the
tls: key is different so this will be added without overwriting anything.
kustomization.yaml also features a
configurations specification which points to a YAML file telling Kustomize how to recognise when other resources are named in cert-manager custom resources. For example, a Certificate definition references an Issuer. If Kustomize adds a name prefix to all resources, then it also needs to add that prefix to references to those resources in the manifests.
Note that Kustomize does not install cert-manager itself, so before applying the cert-manager manifests make sure your cluster has cert-manager installed.
Modify the overlay’s manifests to use the domain you set up earlier, and enter you email address into the Issuer. Apply the cert-manager overlay like so:
kubectl apply -k overlays/cert-manager
Once the resources are deployed cert-manager should fetch a Certificate from Let’s Encrypt staging. When visiting the page it will now use HTTPS. It may give a warning about the Certificate not being trusted, this is just because it is from Let’s Encrypt’s staging environment, so the warning can be ignored.
The development overlay is designed to show how Kustomize can help with multi environment deployments. It uses the cert-manager overlay as a base, which in turn builds upon the Helloweb app.
bases: - ../cert-manager commonLabels: app: helloweb-development namespace: helloweb-development nameSuffix: -development patchesStrategicMerge: - ingress.yaml - issuer.yaml - certificate.yaml resources: - selfsigned-issuer.yaml - selfsigned-certificate.yaml
It overrides the Namespace and
app label, setting them to
helloweb-development. It also adds a
development to all resources. When deployed, these resources will be created from scratch in their own Namespace, so this can be done alongside a deployment of the existing resources.
Certificates must often be issued differently in different environments, so Kustomize is a great way to manage these variations. The development overlay add a self signed Issuer and Certificate, and changes the Issuer and Certificate used by the Helloweb app to use these.
apiVersion: certmanager.k8s.io/v1alpha1 kind: Issuer metadata: name: issuer spec: ca: secretName: ca-secret-development acme: null
This patch to the Issuer tells it to use the CA created by the self signed Issuer and Certificate that have been added. Importantly it also set the
acme: key to
null to remove the values inherited from the base, so that the Issuer can use the
ca: definition instead.
In general Kustomize is only additive, and by design does not allow manifests to be removed, however the patching can be used to set parts of a manifest to
null if they are no longer required.
Once again, modify the overlay’s manifests to use your own domain. It’s suggested that a subdomain like
dev. is used. The overlay can then be deployed like before:
kubectl apply -k overlays/development
Because of the addition of a name suffix new resources will be created in a new namespace. Observe the new Ingress resource to find it’s external IP address and update the chosen domain’s record as before. When the Certificate is issued you can access the page using HTTPS. It will also give a warning, but inspecting the Certificate will show it was signed by the self signed CA.
Staging is another overlay based on the cert-manager overlay and designed to exist alongside the development overlay.
bases: - ../cert-manager commonLabels: app: helloweb-staging namespace: helloweb-staging nameSuffix: -staging patchesStrategicMerge: - ingress.yaml - issuer.yaml - certificate.yaml secretGenerator: - name: ca-secret files: - secret/tls.crt - secret/tls.key type: "kubernetes.io/tls"
It is similar to develop, overriding the
app label, Namespace and adding a different suffix. However, it changes the Issuer and Certificate to use a CA that is locally generated. This makes use of Kustomize’s Secret generator. A TLS key and Certificate can be created locally and used in the
This overlay can be deployed and set up in the same way as the development overlay.
Production is the final overlay, it also uses the cert-manager overlay as a base. Again, it can be deployed like the previous two overlays with it’s own
app label, Namespace and name suffix. It’s effect is more minimal, just changing the Let’s Encrypt ACME provider to the production version, rather than staging, to get a fully valid Certificate.
This overlay can also be deployed and set up like the previous ones. However as it requests a production valid Certificate there should be no warning when accessing it over HTTPS.
Finally, the last overlay is multi-environment. It shows the multi base feature of Kustomize. It does not perform any modifications, but by giving the development, staging and production overlays as bases it allows them to all be generated and applied at once.
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../development - ../staging - ../production
Kustomize is a very flexible and helpful tool. It’s layered approach to managing manifests makes is very well suited to managing your own variations of other peoples work, and to managing variations across environments, without requiring duplication.
As this demo shows, Kustomize makes it easy to start with a minimal app deployment, as is often provided with an app, then modify and build on this to suit your own requirements. It can also be used to separate parts of a deployment so they can be more easily worked on by different teams. All of these modifications are expressed explicitly, making it clear what the changes are, and allowing them to be put under version control.