The Best Secrets Management Tool for Kubernetes
Welcome to Develeap’s Woman Tech Hour! Every two weeks, a group of super-professional DevOps experts at our company gather to discuss and share our insights and experiences on various technical topics. We believe in learning from each other and growing together, and now we want to share our discussion with you. Dive in and benefit from our collective expertise just as we have!
What We Discussed Today
Our discussion centered around the most secure way to handle secrets in Kubernetes. We exchanged experiences about how secrets are managed across our different customer’s environments, explored the details and differences between various tools, and even tackled some common fears head-on.
And yes, we concluded with a clear recommendation!
Why Are Secrets a Problem?
Kubernetes includes a native ‘secret’ resource designed to store sensitive information encoded in base64. However, this approach is insufficient, primarily because Infrastructure as Code (IAC) practices mean these secret resources are committed to git repositories. Therefore to maintain security, secrets should be stored in a secure vault outside the cluster and version control systems. This raises the obvious question: How do we securely transfer these secrets into Kubernetes, and where are the secrets stored inside the cluster?
Secret Management tools
There are numerous tools available to pull secrets from an external vault into a Kubernetes cluster, like External Secrets Operator (ESO), HashiCorp Vault Kubernetes Operator, Azure Key Vault Provider, Google Cloud Secret Manager CSI Driver and more. The choice of tool often depends on where you are running your Kubernetes. If you are an AWS user, you might want to check out these 2 great articles – How to securely and easily pass secrets to the cluster? and Declarative Configuration of Repository Credentials for ArgoCD, Using External Secrets Operator – written by develeap’s experts, that detail the configuration and usage of different solutions for EKS clusters.
Through sharing the tools we use at our different customers, we concluded that all these tools perform the same basic function: they pull secrets from an external vault and provide access to them inside the cluster. However, the main difference is in the way these secrets are stored in the cluster.
Where Are Secrets Saved in the Cluster?
There are two main ways to store secrets within the Kubernetes cluster:
- Creating Secret Resources:
The operator creates native Kubernetes ‘secrets’ and syncs the values from the vault to the ‘secret’ resource.
- Injecting Secrets as Environment Variables:
The operator injects the secret directly into the pod as an environment variable. This method avoids creating secret resources in the cluster altogether.
Is it Safe to Store Secrets in a ‘Secret’ Resource?
A few of us shared that they have encountered reluctance from colleagues and managers to use tools that create secrets within the cluster, believing it’s not safe enough. Many people and companies prefer using injection because it feels safer – what’s the point of having the secrets stored in a secure vault if at the end they can all be listed from inside the cluster. However, if you think about it, if a malicious actor gains access to the cluster, they can exec into the pod and read the environment variables as easily as listing secret resources.
Therefore, secrets management must be paired with robust access management!
By securing your cluster through networking and Role-Based Access Control (RBAC), it is more than safe to use native Kubernetes secrets. Companies go out of their way to have no ‘secret’ resources in the cluster, and pay the price of more complex configurations and harder access for developers and customers who actually need access to those secrets, as they are “hidden” inside the pods.
How to Stop Being Afraid of the Word RBAC
An interesting point that came up in our discussion is that many people, even experienced DevOps, get intimidated by the term “RBAC.” This fear often leads companies to neglect access management and rely solely on secrets management for security. Consequently, there’s a perception that secret operators which create secrets within the cluster aren’t secure enough, leading to the search for more complex solutions. But here’s the thing—if your secrets aren’t stored in plain text and you’re using a secure vault with an operator to manage those secrets in the cluster, you’re on the right track. It’s better to focus your efforts on securing firewalls and applying strong RBAC controls than worrying that your operator isn’t secure enough.
Bottom Line
Needless to say, you should store all of your secrets in a secret vault outside of source control. For pulling the secrets into the cluster, any secrets management operator that is convenient for you should be good enough – even if it stores that secret in a “plain” secret resource – provided that it is combined with proper RBAC control. The issue should not be which operator you are using, as most are likely good enough; instead, you should focus more on implementing robust RBAC.
RBAC is a crucial topic that deserves its own dedicated discussion. We hope to delve deeper into RBAC in a future meeting, exploring its intricacies and best practices. Until then, check Ori Meyuhas great article Managing EKS authentication with IAM groups and assigning a RBAC to the group, that has a detailed guide on managing group-level access to a Kubernetes cluster using AWS IAM and Kubernetes RBAC to ensure granular, scalable, and manageable permissions for users.
We hope you gained knowledge and confidence from this article, whether you’re searching for the right tool for your company or weren’t confident enough that the tool you are currently using is secure enough. Stay tuned for more insights from our Woman Tech Hour sessions.