Kubernetes External Secrets Operator
External Secrets Operator is an open source project targeting a common issue which is managing Kubernetes secrets (but not only!) in a GitOps yet cloud agnostic way. Its Kubernetes controller will monitor any ExternalSecret manifest applied to our cluster and leverage our secrets management system configuration to link, generate, and update Kubernetes secrets manifests needed for our deployments.
The Operator can integrate with many secret management solutions:
- HashiCorp Vault
- Azure Key Vault
- AWS Secrets Manager
and many more.
The Problem:
Secrets are a substantial part of our applications, but their management could be a hassle. We’d like to follow GitOps principles, yet we can’t have them plain-text stored in our Git repository or even base64 encoded as Kubernetes secrets are. We’re already in the cloud, and would like to leverage our clouds’ secrets management system, however, we may extent to a multi-cloud deployment in the future.
Demo: External Secrets Operator to the rescue!
To demonstrate the strong features of using this operator, we’ll show you a quick and easy guide to getting started, and work our magic from a cloud stored secret, to an up-to-date kubernetes secret.
we will be using our favorite cloud — AWS:
- EKS as our Kubernetes solution.
- SecretsManager as our secrets management solution.
- KMS to encrypt our secrets at rest and in transit.
- BONUS: SSMParameterStore! Our operator integrates great with AWS Parameter Store as well, allowing better convenience for our developers.
Permissions
Since we’re top-notch Security Specialist Professionals, we start by creating an IAM Role in our AWS account, which will later be attached to our Operators’ service account. The process is also known as using IRSA which stands for IAM Role for Service Accounts.
This approach takes advantage of short-lived credentials instead of persistent AWS credentials, and manages permissions using AWS policies.
Create an IAM Policy
The following policy will give our future role permissions to decrypt the different versions of secrets stored in secrets manager, and will also work with SSM parameter store (bonus!).
- In this example, we only allow a single course of action, which is to manage secrets in the cloud ONLY. However the operator is capable of updating secret values from within kubernetes — NOT RECOMMENDED.
“external-secrets-operator-demo-policy”:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccessToSecretsManager",
"Effect": "Allow",
"Action": [
"secretsmanager:ListSecrets",
"secretsmanager:GetSecretValue",
"secretsmanager:DescribeSecret",
"secretsmanager:ListSecretVersionIds"
],
"Resource": [
"arn:aws:secretsmanager:us-east-1:123456789012:secret:*"
]
},
{
"Sid": "AllowAccessToParameterStore",
"Effect": "Allow",
"Action": [
"ssm:GetParameters",
"ssm:GetParameter",
"ssm:GetParametersByPath"
],
"Resource": [
"arn:aws:ssm:us-east-1:123456789012:parameter/*"
]
},
{
"Sid": "AllowAccessToKMSDefaultKey",
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": [
"arn:aws:kms:us-east-1:123456789012:key/02e225ec-2045-4811-bd8b-05a2a90f53bc"
]
}
]
}Create an IAM Role
We will name this role “external-secrets-operator-demo-role” and attach the “external-secrets-operator-demo-policy” to it.
Also, we’ll make sure to set the Trust Policy appropriately, so our Kubernetes Service Account for the next steps, will be able to assume this role successfully.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "TrustMyRole",
"Effect": "Allow",
"Principal": {
"Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/FCEDA5370A14B25672159FC1FE731DFC"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"oidc.eks.us-east-1.amazonaws.com/id/FCEDA5370A14B25672159FC1FE731DFC:aud": "sts.amazonaws.com",
"oidc.eks.us-east-1.amazonaws.com/id/FCEDA5370A14B25672159FC1FE731DFC:sub": "system:serviceaccount:demo:external-secrets-operator-demo-service-account"
}
}
}
]
}IAM Role with Policy
Install the Operator
The Operator can be installed in many different way, direct YAML install, Helm chart installation, Terraform module, etc.
My absolute favorite is using the notorious aws-ia/eks-blueprints-addon Terraform module with a Terragrunt wrapper.
However, for simplicity, we’ll just use plan helm chart installation.
Let’s define our precious values in “values.yaml” first:
fullnameOverride: "external-secrets"
clusterName: external-secrets-operator-demo
clusterEndpoint: ${dependency.eks.outputs.cluster_endpoint}
serviceAccount:
name: external-secrets-operator-demo-service-account
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam:us-east-1:Z1234567890:role/external-secrets-operator-demo-roleNext, we can install the chart while feeding it our values file from previous step.Quick tip: it’s best to work with explicit versions, as breaking changes might occur between different versions. For demo purposes only, we’ll use latest (‘helm-chart-0.9.4’ as of today).
helm repo add external-secrets https://charts.external-secrets.io
helm install --namespace demo --values ./values.yaml external-secrets external-secrets/external-secretsDeploy a ClusterSecretStore
You could either deploy a SecretStore which is namespace scoped, or a ClusterSecretStore .
I like having a dedicated namespace for each controller / application / service, so i prefer having a centralized secret store.
To begin, we write our ClusterSecretStore manifest.
Since our operator works great with AWS SSMParameterStore as well, we might as well allow our developers to leverage both.
kind: ClusterSecretStore
metadata:
name: aws-secrets-manager
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-operator-demo-service-account
namespace: demo
---
kind: ClusterSecretStore
metadata:
name: aws-parameter-store
spec:
provider:
aws:
service: ParameterStore
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets-operator-demo-service-account
namespace: demoLet’s apply our manifests locally using `kubectl`:
kubectl apply — filename cluster-secret-stores.yamlValidate our ClusterSecretStore s status using:
kubectl get clustersecretstores
Create our first ExternalSecret
We start of by creating a new secret in AWS Secrets Manager. Named external-secrets-operator-demo.
aws secretsmanager create-secret \
--name external-secrets-operator-demo \
--description "External Secrets Operator Demo." \
--secret-string "{\"foo\":\"bar\",\"password\":\"EXAMPLE-PASSWORD\"}"Set the encryption:
- we are using the default kms/ssm encryption key for demonstration purposes.
Write our “external-secret-manifest-demo.yaml” manifest
## Create an external secret lined to AWS Secrets Manager secret
## This will generate a kubernetes secret with the named "my-kubernetes-secret" in the demo namespace
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: my-external-secret
namespace: demo
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: my-kubernetes-secret # Secret name in k8s
creationPolicy: Owner
## You can get all of the secret content (recommended)
dataFrom:
- extract:
key: my-cloud-secret
## Or you can get specific fields
data:
- secretKey: my-kubernetes-secret-field # Name of the field to store inside the k8s secret
remoteRef:
key: my-cloud-secret # Our AWS SecretsManager secret-name goes here
property: password # Retrieve the password field- It’s best practice to follow GitOps principles, and store this manifest in your gitops repository, which is monitored by ArgoCD.
Just for demonstration purposes, we’ll apply manually.
kubectl apply — filename external-secret-manifest-demo.yamlValidate our ExternalSecret status using:
kubectl get externalsecret my-external-secret — namespace demoKubernetes secret
Let’s see the generated Kubernetes secret created for us:
kubectl get secret -n demoKubernetes secret
We are all set!
