Kubernetes Secrets Management
This is our second article in the “Kubernetes Security series”. This article talks about Secret Management in Kubernetes.
Once you start deploying applications to Kubernetes, you quickly realize Secrets Management takes a lot of time to get right. The Kubernetes Secret and ConfigMap objects don’t yet have all the features you’d expect to properly setup Secrets Management. Until Kubernetes has these capabilities built-in, you’ll have to come up with your own mechanism to deal with secrets.
There are a lot of options to chose from. It all depends on what kind of setup and cloud provider you are using. Here are a few options that I’ll explain in more detail in this article:
Using the Kubernetes Secrets / ConfigMap objects Using Kubernetes Secrets / ConfigMap, but leveraging encryption capabilities in conjunction with AWS KMS, Azure Vault, or other key management systems Using a Cloud integration, for example AWS Secrets, AWS Parameter Store, or Azure Vault A third party like Hashicorp Vault, to manage secrets Your own custom solution using MutatingWebhooks All these different approaches have one thing in common. You don’t want to store your secrets unprotected in a system that wasn’t made to store secrets. You shouldn’t store secrets in the code, nor in the “Infrastructure as Code” repositories that you use for tooling like terraform or AWS CloudFormation. Secrets should be managed in a way that only authorized users/applications can have access to it. Audit trails should be in place, and encryption at rest / on the wire should always be present when dealing with secrets.
Kubernetes Secrets / ConfigMap
Kubernetes itself provides you with the Secrets and ConfigMap objects. You can use these out-of-the-box just like you use other objects in Kubernetes like Deployments and Services. You can easily refer to a ConfigMap or a Secret from another object, as these are all are native objects in the Kubernetes cluster. They also tie into Kubernetes RBAC, ensuring that only authorized users and applications can read the information in Secrets / ConfigMaps. Sounds great so far!
There’s only one major drawback. The contents of these objects will be stored on the Kubernetes backend database, etcd, in plain text (by default), as shown in above diagram. Anyone having access to the Kubernetes master will be able to see the secrets by looking it up in the backend.
Conclusion: just using Secrets objects (or storing secret information in ConfigMaps) without any other way to secure the secrets, is probably not going to be enough for your organization to have them securely stored.
Encrypt the secrets!
We can easily solve the problem of unencrypted secrets… by encrypting them! This sounds like an easy solution, but to encrypt data, you need encryption keys, and you need to manage them, which in turn can be complex and cumbersome.
If you’re on a public cloud provider, Key Management Services can make your life (a lot) easier. On AWS EKS for example, KMS can tie in directly with the Kubernetes Secrets, using envelope encryption and storing the master key in KMS, as shown in the diagram below.
If you’re not on AWS or you don’t have a direct integration available on your Kubernetes cluster, you can still store your encryption keys in a Key Management System. Azure Vault has this capability, but it can be any Key Management System (preferably one that uses a Hardware Security Module – HSM – to protect the key material). It’ll be up to you to design and create the necessary scripts or integrations to ensure encryption and decryption happens in a secure way. The Kubernetes Operator pattern can help you with providing the framework for this management of secrets to happen. An example flow is shown in the diagram below.
Cloud Integrations
Public Cloud providers already have a way to manage secrets. Kubernetes can be extended in different ways to plug in to these external storage systems, including for secrets. There are 2 common approaches:
Using MutatingWebhooks to intercept Kubernetes object creation, then the secrets can be added transparently by interacting with external providers Mounting the secrets by using the Container Storage Interface (CSI) driver A good example of the MutatingWebhook is AWS’s example implementation of the AWS Secrets Manager. On their blog they explain how a controller could use the MutatingWebhook capabilities to inject secrets into the pod specification.
Azure’s integration for AKS with Azure Vault uses the Container Storage Interface (CSI). Once enabled the pod specification can include a volume definition to mount secrets as files in the container.
HashiCorp Vault
A more generic solution is to use HashiCorp’s Vault, which is a third party software product that you can use to store your secrets. The HashiCorp implementation uses MutatingWebhooks to inject an init container and a sidecar. For a correctly annotated pod, the init container will fetch the secrets, and the sidecar will keep them up-to-date, checking if the secret hasn’t changed in HashiCorp Vault. The application in the container can then read the files on the filesystem that contain the secrets.
This solution sounds like an easy way to implement secrets, but HashiCorp Vault is a 3rd party software product and comes with its own maintenance tasks that shouldn’t be overlooked. One company that uses HashiCorp Vault, GoCardless, experienced a full service outage because of an expired TLS certificate that made it impossible to connect to Vault. That’s just saying it’s a very critical component that you need to fully understand before bringing it onboard in your organization to handle your secrets management.
Custom Solution
Now that we’ve gone over the different implementations, you can see that implementing your own solution wouldn’t be very difficult and can be done using a similar approach. The MutatingWebhook has my personal preference, because it’s a very transparent solution. There is no need to rewrite your app to read secrets from the filesystem, or changing your pod specification to add volume mounts. Once annotations are in place, the MutatingWebhook can take care of the rest, as shown in the diagram below.
When using option 1, you’ll need to encrypt your secrets within the Secrets object. The MutatingWebhook will need to inject an init container and a sidecar which will contain the logic to decrypt the secrets. Any Key Management System can be plugged in here, as you’ll be responsible to write the code doing the actual decryption.
With option 2, you don’t use the Secret object, but rather an external secret store. The solution is similar to Hashicorp Vault, but it can really be any generic secret store. The init container and sidecar will interface the secret store to inject the correct secrets in the actual app.
Secrets Management – much broader than what I discussed
Secrets Management is not easy. The management of secrets within your Kubernetes cluster is only a small part of the overall secrets management process. You’ll also have to make sure that you don’t have secrets lying around at any point of the deployment of the application. That means no secrets bundled with the app, no secrets in git, no secrets in terraform state files, no secrets in any system that can’t provide you with sufficient controls to protect them. At IN4IT, we implement custom solutions using operators written in Go. Those controllers can transparently handle secret injection, without the developer having to change their code.
