We recently worked on a customer project, where they wanted to secure the connection between their Java Spring Boot application and their MySQL Database, all this running on Google Kubernetes Engine (GKE). We suggested they use cert-manager, our preferred certificate management tool on Kubernetes.
It was the first time that we would use cert-manager to secure MySQL for a customer project and during the deployment of it we encountered some challenges.
In this post, we will show you how to secure the connection between a Java Spring Boot application and MySQL in a Kubernetes environment with cert-manager.
Deploy cert-manager
What if someone intercepted sensitive data that is stored in your database? That is why it is important to add TLS (it is common to use the term SSL but today usually refers to TLS) encryption between your app and the MySQL server. To do so, we need to configure MySQL to use encrypted connections using TLS certificates provisioned by cert-manager.
Install all cert-manager’s resources and CustomResourceDefinitions. We do it using regular manifests for Kubernetes, but you can find more ways of installation here.
# Kubernetes 1.15+
kubectl apply --validate=false -f https://github.com/jetstack/cert-manager/releases/download/v0.15.0/cert-manager.yaml
You can verify it is installed correctly by checking the cert-manager pods running:
kubectl get pods --namespace cert-manager
Install Issuers
Now that we have the certificate management tool deployed correctly, let’s get the Issuers which will represent certificate authorities (CAs) that are able to generate signed certificates. In this case we will setup a CA and SelfSigned issuer.
An Issuer is a namespaced resource, so you will need to create the issuers in the same namespace as your application and database resources. You can create a ClusterIssuer instead if you want to be able to request certificates from any namespace.
SelfSigned Issuer
This is useful to generate a root CA for use with the CA Issuer.
Create this manifest locally and apply it to your cluster.
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: selfsigned-issuer
spec:
selfSigned: {}
$ kubectl apply -f selfsigned-issuer.yaml
issuer.cert-manager.io "selfsigned-issuer" created
CA Issuer
We’re generating an internal certificate authority (CA) that will be used to sign incoming certificate requests for the MySQL instance. Both MySQL server and the client will rely on the CA to to make sure a veritable connection.
Certificate CA
In order to create the CA issuer, you must first create a self signed CA, which will be issued by issuer selfSigned.
Note that isCA is set to true in the body of the spec.
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: ca-certificate
spec:
secretName: ca-cert
duration: 2880h # 120d
renewBefore: 360h # 15d
commonName: MySQL admin
isCA: true
keySize: 2048
usages:
- digital signature
- key encipherment
issuerRef:
name: selfsigned-issuer
kind: Issuer
group: cert-manager.io
Copy the manifest above and apply it:
$ kubectl apply -f ca-certificate.yaml
Note: The Secret resource will contain the certificate and signing key, this will be created when the CA certificate is deployed, and the CA issuer references that secret. So that it will trust resulting signed certificates.
Next step is to deploy the CA issuer.
apiVersion: cert-manager.io/v1alpha2
kind: Issuer
metadata:
name: ca-issuer
spec:
ca:
secretName: ca-cert
$ kubectl apply -f ca-issuer.yaml
issuer.cert-manager.io "ca-issuer" created
You can then check that the issuers have been successfully configured by checking the status, example:
$ kubectl get issuers
NAME READY STATUS AGE
ca-issuer True Signing CA verified 60s
MySQL TLS and client certificates authentication
Once the Issuers are deployed, you are ready to request your certificates. It’s important to set the correct usages, otherwise the certificate will be created incorrectly (we had this issue, which is why we’re writing this post so you can avoid the same mistake).
Most of the certificates require by default the usages digital signature and key encipherment. We will add server auth to the server certificate and client auth to the client to ensure that the certificates can be used for client/server authentication.
The signed certificates will be stored in a secret resource in the same namespace as the certificates.
MySQL Server Certificate
Let’s start by requesting the MySQL server certificate and private key, copying the following manifest and applying to your cluster with the command kubectl apply -f mysql-server.yaml
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: mysql-server
spec:
secretName: mysql-server-cert
duration: 2160h # 90d
renewBefore: 360h # 15d
isCA: false
keySize: 2048
keyAlgorithm: rsa
keyEncoding: pkcs1
usages:
- digital signature
- key encipherment
- server auth
commonName: MySQL server
issuerRef:
name: ca-issuer
kind: Issuer
group: cert-manager.io
I guess you are wondering how MySQL will be able to use the certificate. Let’s suppose your MySQL server is running as a pod in your Kubernetes environment. Create a ConfigMap with your MySQL Server configuration file:
apiVersion: v1
kind: ConfigMap
metadata:
name: mysql-config
data:
mysql.cnf: |-
[mysqld]
ssl-ca=ca.crt
ssl-cert=tls.crt
ssl-key=tls.key
require_secure_transport=ON ## This line is the only setting required to enforce secure connections
Add this ConfigMap data and the Secret which contains the cert and key of the certificate created previously to a volume inside your MySQL pod or deployment. Now TLS is enabled on your MySQL server.
Access your MySQL server and check the values of the TLS related variables have been populated with the names of the certificates that we generated:
SHOW VARIABLES LIKE '%ssl%';
# Output
+---------------+-----------------+
| Variable_name | Value |
+---------------+-----------------+
| have_openssl | YES |
| have_ssl | YES |
| ssl_ca | ca.crt |
| ssl_capath | |
| ssl_cert | tls.crt |
| ssl_cipher | |
| ssl_crl | |
| ssl_crlpath | |
| ssl_key | tls.key |
+---------------+-----------------+
9 rows in set (0.00 sec)
MySQL Client Certificate
Currently, we have MySQL server’s certificate signed by certificate authority (CA) and a key pair. Having these is enough to provide encryption for incoming connections.
However this is not enough for us, as we also need to authenticate the clients connecting to our MySQL server. To do so, we will request a client certificate and key, so that both parties can provide proof that their certificates were signed by a mutually trusted certificate authority.
MySQL TLS Connection Using JDBC
MySQL Connector/J is the Java library that manages the connection with the MySQL database. It can use TLS certificates to encrypt all that data between the JDBC driver and the MySQL server.
MySQL Connect/J has different ways of setting up an TLS connection. We are setting up two-way TLS authentications, in this form, both the client and the server has to establish trust between themselves using a trusted certificate.
Java application needs two Java keystore files to communicate over TLS. The truststore one file which contains CA certificate, and another called keystore which contains the keys and certificate for the client.
Java’s keytool is used to import CA certificate into Java truststore file, and import the client key and certificate into a Java keystore.
The latest version of cert-manager can do this for you, as you can see on the manifest, we are adding the keystore option on the Certificate spec. The keystore will be added in the Secret resource.
First, create a secret containing the keystore password, which will be used by cert-manager to generate the truststore.
kubectl create secret generic jks-password-secret --from-literal=password-key=[your-password]
Let’s create that client certificate and private key, same as you did with the server certificate, but using this manifest:
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
name: client-mysql
spec:
secretName: mysql-client-cert
duration: 2160h # 90d
renewBefore: 360h # 15d
keySize: 2048
keyAlgorithm: rsa
keyEncoding: pkcs1
usages:
- digital signature
- key encipherment
- client auth
commonName: MySQL client
keystores:
jks:
create: true
passwordSecretRef:
key: password-key
name: jks-password-secret
issuerRef:
name: ca-issuer
kind: Issuer
group: cert-manager.io
Mount the secret as volume on your app deployment, so that the keystore and truststore will be available for the application. You can get more details about secrets and how to create them from the Kubernetes documentation.
Edit your app deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
...
spec:
replicas: 1
selector:
...
template:
...
spec:
containers:
- name: app
image: your-app-image
volumeMounts:
- mountPath: "/certs"
name: truststore
readOnly: true
- mountPath: "certs2"
name: keystore
readOnly: true
...
volumes:
- name: truststore
secret:
secretName: mysql-client-cert
items:
- key: truststore.jks
path: truststore
- name: keystore
secret:
secretName: mysql-client-cert
items:
- key: keystore.jks
path: keystore
Finally, update your application’s connection string, adding the useSSL, trustCertificateKeyStoreUrl, trustCertificateKeyStorePassword, clientCertificateKeyStoreUrl and clientCertificateKeyStorePassword parameters.
The connection string should look something like this:
jdbc:mysql://[url]:[port]/[dbname]?useSSL=true&clientCertificateKeyStoreUrl=file:/certs/keystore&clientCertificateKeyStorePassword=[your-password]&trustCertificateKeyStoreUrl=file:/certs/truststore&trustCertificateKeyStorePassword=[your-password]
The trustCertificateKeyStorePassword and clientCertificateKeyStorePassword keys set the password value that you set when you created the secret (we called it as jks-password-secret in this post) which is used by cert-manager to generate the java keystores.
Hopefully this post will be useful for your Java Spring Boot implementation, but most importantly for the security of your database.