cert-manager has become the de facto solution for managing X.509 certificates for applications running in Kubernetes. The project, now part of the CNCF Sandbox, was built with flexibility and extensibility in mind. It allows operators to use Certificates and Issuers as first-class citizens in the API, and for this to enable almost any use case in which an automated certificate is required.
As Kubernetes adoption grows, there is now increased interest in service mesh to securely connect workloads, as well as provide dynamic configuration of traffic routing and consistent observability. Services meshes typically use X.509 certificates for mutual TLS (mTLS), with the identity encoded using SPIFFE.
With a range of supported certificate issuers, there has been interest from the community to see Istio integrated with cert-manager for it to provide signing capabilities for workloads. Working with the Security WG in the Istio community, as well as a number of our customers, the cert-manager team at Jetstack have built an integration that enables cert-manager to sign workload certificates in an Istio service mesh.
Rise of the service mesh
Istio is a popular, fully-featured service mesh; it has a rich set of configurations for traffic routing, policy control, and observability. Like most service meshes, the data plane is powered by Envoy, and each workload running in the mesh will have one of these proxies that intercept all traffic, and make routing decisions based upon the mesh configuration as a whole.
When it comes to managing certificates, service meshes typically ship with a limited number of Certificate Authority (CA) choices, those being either self signed, perhaps Vault, and sometimes an integration with a cloud provider solution. This imposes limitations on how and where your certificates get signed. These limitations can be problematic for larger enterprises where there are requirements for using particular CAs, auditing and visibility of these signed certificates, as well as the headaches that come with changing the Certificate Authority already used in production.
We’ve gone deeper into Istio OIDC Authentication here.
Why use cert-manager?
cert-manager is commonly used to secure ingresses, in conjunction with an Ingress controller (NGINX, for example) and a public CA such as Let’s Encrypt. You can use cert-manager with Istio today to secure ingress using the Istio Gateway (guide), but up until now it’s not been straightforward to use for issuance and renewal of workload certificates.
cert-manager was always built to be CA-agnostic and enable integration with both public and private CAs. The project and wider ecosystem has a wide choice of public and private CA providers available for certificate signing. There are internal issuers core to the project, including HashiCorp Vault, and a fast-growing set of External Issuers that should cover many bases, including cloud provider PKI services. It’s also simple enough to build on cert-manager and develop your own external issuer if you’ve a more bespoke requirement.
A further benefit of using cert-manager is its centralised API and consistent interface. This means that swapping out an issuer, say to use a different CA provider, is fairly trivial. This operation only requires a change to the Issuer configuration and issuer that the mesh is configured to use - the rest is all handled by cert-manager.
By using cert-manager as the single tool for managing certificates in the cluster, operators are able to have complete visibility of the machine identities with a single pane of glass, for auditing, policy and observability purposes (and if that sounds useful, check out our Jetstack Secure service - it’s free to get started).
Machine identity lifecycle in an Istio service mesh
When a workload starts, along with its sidecar, it needs to identify itself to
the mesh control plane, and retrieve a signed certificate in order to begin
sending traffic over the network. A helper executable called the
responsible for retrieving this signed certificate, and delivering it to Envoy to use.
This involves sending a certificate signing request to the control
plane component of Istio (
istiod), along with the pod’s Service Account token
to securely identity itself. Once
istiod has validated the request, it will in
the default case, sign and send back a certificate trusted by the mesh that
Envoy can begin to serve and send traffic.
How we’ve integrated cert-manager
The cert-manager integration with Istio involves running a separate
cert-manager-istio-agent processes certificate requests from istio-agents in the mesh.The agent will verify the identity, using the incoming Service Account token and ensure the
certificate signing request matches that identity, kicking off a
CertificateRequest flow with cert-manager. This involves cert-manager watching
CertificateRequest created, and signing it using the Issuer installed
and referenced on the request. Once signed, the certificate is returned back to
istio-agent to facilitate mTLS in the mesh.
This agent is responsible for distributing the trust chain of the configured Issuer, along with signing certificates for all workloads running in the cluster.
istio-csr project, you can integrate any of cert-manager’s Issuers and have this
provide certificate signing for workloads in the mesh. With more External
Issuers, such as the
Google CAS Issuer, being added
to the cert-manager ecosystem all the time, it allows certificates in all areas
of the cluster be signed from any number of centralised CAs that best suits
your needs, whilst ensuring the portability that the cert-manager API provides.
Not using Istio?
In the CNCF, the Linkerd project already provides cert-manager integration. Jetstack has also integrated Open Service Mesh (OSM), a new service mesh from Microsoft in the CNCF Sandbox - (see this video for a talk featuring a demo).
If you are interested in trying out the new Istio cert-manager integration, you can find the repository and documentation on GitHub here. Let us know what you think! We’re here to help should you want to discuss further integration and find out more about Jetstack Secure for cert-manager.