External Secrets Operator is an open source project targeting a common issue which is managing Kubernetes secrets (but not only!) in a GitOps yet cloud agnostic way. Its Kubernetes controller will monitor any ExternalSecret manifest applied to our cluster and leverage our secrets management system configuration to link, generate, and update Kubernetes secrets manifests needed for our deployments.

The Operator can integrate with many secret management solutions:

• HashiCorp Vault
• Azure Key Vault
• AWS Secrets Manager

and many more.

The Problem:

Secrets are a substantial part of our applications, but their management could be a hassle. We’d like to follow GitOps principles, yet we can’t have them plain-text stored in our Git repository or even base64 encoded as Kubernetes secrets are. We’re already in the cloud, and would like to leverage our clouds’ secrets management system, however, we may extent to a multi-cloud deployment in the future.

Demo: External Secrets Operator to the rescue!

To demonstrate the strong features of using this operator, we’ll show you a quick and easy guide to getting started, and work our magic from a cloud stored secret, to an up-to-date kubernetes secret.

we will be using our favorite cloud — AWS:

• EKS as our Kubernetes solution.
• SecretsManager as our secrets management solution.
• KMS to encrypt our secrets at rest and in transit.
• BONUS: SSMParameterStore! Our operator integrates great with AWS Parameter Store as well, allowing better convenience for our developers.

Permissions

Since we’re top-notch Security Specialist Professionals, we start by creating an IAM Role in our AWS account, which will later be attached to our Operators’ service account. The process is also known as using IRSA which stands for IAM Role for Service Accounts.

This approach takes advantage of short-lived credentials instead of persistent AWS credentials, and manages permissions using AWS policies.

Create an IAM Policy

The following policy will give our future role permissions to decrypt the different versions of secrets stored in secrets manager, and will also work with SSM parameter store (bonus!).

• In this example, we only allow a single course of action, which is to manage secrets in the cloud ONLY. However the operator is capable of updating secret values from within kubernetes — NOT RECOMMENDED.

“external-secrets-operator-demo-policy”:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowAccessToSecretsManager",
            "Effect": "Allow",
            "Action": [
                "secretsmanager:ListSecrets",
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecretVersionIds"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-1:123456789012:secret:*"
            ]
        },
        {
            "Sid": "AllowAccessToParameterStore",
            "Effect": "Allow",
            "Action": [
                "ssm:GetParameters",
                "ssm:GetParameter",
                "ssm:GetParametersByPath"
            ],
            "Resource": [
                "arn:aws:ssm:us-east-1:123456789012:parameter/*"
            ]
        },
        {
            "Sid": "AllowAccessToKMSDefaultKey",
            "Effect": "Allow",
            "Action": [
                "kms:Decrypt"
            ],
            "Resource": [
                "arn:aws:kms:us-east-1:123456789012:key/02e225ec-2045-4811-bd8b-05a2a90f53bc"
            ]
        }
    ]
}

Create an IAM Role

We will name this role “external-secrets-operator-demo-role” and attach the “external-secrets-operator-demo-policy” to it.

Also, we’ll make sure to set the Trust Policy appropriately, so our Kubernetes Service Account for the next steps, will be able to assume this role successfully.

{
    "Version": "2012-10-17",
    "Statement": [
        {
      "Sid": "TrustMyRole",
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::123456789012:oidc-provider/oidc.eks.us-east-1.amazonaws.com/id/FCEDA5370A14B25672159FC1FE731DFC"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "oidc.eks.us-east-1.amazonaws.com/id/FCEDA5370A14B25672159FC1FE731DFC:aud": "sts.amazonaws.com",
                    "oidc.eks.us-east-1.amazonaws.com/id/FCEDA5370A14B25672159FC1FE731DFC:sub": "system:serviceaccount:demo:external-secrets-operator-demo-service-account"
                }
            }
        }
    ]
}

IAM Role with Policy

Install the Operator

The Operator can be installed in many different way, direct YAML install, Helm chart installation, Terraform module, etc.

My absolute favorite is using the notorious aws-ia/eks-blueprints-addon Terraform module with a Terragrunt wrapper.

However, for simplicity, we’ll just use plan helm chart installation.

Let’s define our precious values in “values.yaml” first:

fullnameOverride: "external-secrets"
clusterName: external-secrets-operator-demo
clusterEndpoint: ${dependency.eks.outputs.cluster_endpoint}
serviceAccount:
  name: external-secrets-operator-demo-service-account
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam:us-east-1:Z1234567890:role/external-secrets-operator-demo-roleNext, we can install the chart while feeding it our values file from previous step.

Quick tip: it’s best to work with explicit versions, as breaking changes might occur between different versions. For demo purposes only, we’ll use latest (‘helm-chart-0.9.4’ as of today).

helm repo add external-secrets https://charts.external-secrets.io
helm install --namespace demo --values ./values.yaml external-secrets external-secrets/external-secrets

Deploy a ClusterSecretStore

You could either deploy a SecretStore which is namespace scoped, or a ClusterSecretStore .

I like having a dedicated namespace for each controller / application / service, so i prefer having a centralized secret store.

To begin, we write our ClusterSecretStore manifest.

Since our operator works great with AWS SSMParameterStore as well, we might as well allow our developers to leverage both.

kind: ClusterSecretStore
metadata:
  name: aws-secrets-manager
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-operator-demo-service-account
            namespace: demo
---
kind: ClusterSecretStore
metadata:
  name: aws-parameter-store
spec:
  provider:
    aws:
      service: ParameterStore
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-operator-demo-service-account
            namespace: demo

Let’s apply our manifests locally using `kubectl`:

kubectl apply — filename cluster-secret-stores.yaml

Validate our ClusterSecretStore s status using:

kubectl get clustersecretstores

Create our first ExternalSecret

We start of by creating a new secret in AWS Secrets Manager. Named external-secrets-operator-demo.

aws secretsmanager create-secret \
    --name external-secrets-operator-demo \
    --description "External Secrets Operator Demo." \
    --secret-string "{\"foo\":\"bar\",\"password\":\"EXAMPLE-PASSWORD\"}"

Set the encryption:

• we are using the default kms/ssm encryption key for demonstration purposes.

Write our “external-secret-manifest-demo.yaml” manifest

## Create an external secret lined to AWS Secrets Manager secret
## This will generate a kubernetes secret with the named "my-kubernetes-secret" in the demo namespace
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-external-secret
  namespace: demo
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: my-kubernetes-secret # Secret name in k8s
    creationPolicy: Owner

  ## You can get all of the secret content (recommended)
  dataFrom:
  - extract:
     key: my-cloud-secret

  ## Or you can get specific fields
  data:
  - secretKey: my-kubernetes-secret-field # Name of the field to store inside the k8s secret
    remoteRef:
      key: my-cloud-secret # Our AWS SecretsManager secret-name goes here
      property: password   # Retrieve the password field

• It’s best practice to follow GitOps principles, and store this manifest in your gitops repository, which is monitored by ArgoCD.

Just for demonstration purposes, we’ll apply manually.

kubectl apply — filename external-secret-manifest-demo.yaml

Validate our ExternalSecret status using:

kubectl get externalsecret my-external-secret — namespace demo

Kubernetes secret

Let’s see the generated Kubernetes secret created for us:

kubectl get secret -n demo

Kubernetes secret

We are all set!

Links and references:

• Docs
• ExternalSecrets Operator Official GitHub Repository
• EKS Addon Terraform Module
• Terragrunt
• AWS Secrets Manager
• AWS Parameter Store
• AWS Key Management System