Setup External Secrets Operator at k8s (Vault as the secret backend)

In my homelab I’m trying to keep secrets out of Git while still staying 100% GitOps.

That’s why I’m moving from Sealed Secrets to External Secrets Operator (ESO) + HashiCorp Vault:

• Vault becomes the single source of truth.
• Kubernetes secrets are materialized on-demand by ESO.
• Everything Kubernetes-side (ESO + stores + ExternalSecrets) is explained and versioned in Git, and applied by Argo CD.
• Vault auth/policies/roles are managed as code (Terraform).

This post documents the exact setup I’m using on my Talos Kubernetes cluster.

What I’m building

Goal: ESO authenticates to Vault using a Kubernetes ServiceAccount JWT (TokenRequest) and then reads from Vault KV v2 (e.g. secret/k8s/...) to create Kubernetes Secret objects.

Key details:

• Vault is reachable via HTTPS (valid certificate).
• Vault uses the JWT/OIDC auth method (not auth/kubernetes).
• I pin Kubernetes’ ServiceAccount signing public keys (JWKS → PEM) in Vault so it can validate JWTs without calling Kubernetes discovery endpoints.

High-level architecture

GitOps: installing ESO with Argo CD (multi-source)

I deploy ESO using one Argo CD Application that pulls:

1. the official Helm chart (remote)
2. my custom manifests (ClusterSecretStores, ExternalSecrets, etc.) from a Git repo

This keeps the installation + configuration in one place.

Argo CD Application

Note: this uses spec.sources (plural), not the legacy spec.source.

---
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: external-secrets
  namespace: argocd
spec:
  project: internal
  destination:
    server: https://kubernetes.default.svc
    namespace: external-secrets

  sources:
    # 1) Remote ESO Helm chart
    - repoURL: https://charts.external-secrets.io
      chart: external-secrets
      targetRevision: 1.1.1
      helm:
        releaseName: external-secrets
        values: |
          installCRDs: true
          serviceAccount:
            create: true
            name: external-secrets

    # 2) My manifests repo (ClusterSecretStore, ExternalSecret, etc.)
    - repoURL: https://gitlab.com/vlspr-home/argocd/eso-talos-laughingmanlab-net.git
      targetRevision: main
      path: manifests/external-secrets

  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true

Repo layout (suggested)

manifests/
  external-secrets/
    clustersecretstores/
      vault-minio.yaml
      vault-velero.yaml
      vault-rpi.yaml
    externalsecrets/
      demo-db-secret.yaml

ClusterSecretStore: 3 boundaries, 3 stores

I’m using separate ClusterSecretStores to keep policy boundaries clear:

• vault-minio for MinIO credentials
• vault-velero for Velero backup credentials
• vault-rpi for general/shared secrets (raspberry / misc automation)

They all:

• point to the same Vault server
• use KV v2 at secret/
• authenticate using the ESO ServiceAccount token (external-secrets)

Store: vault-minio

---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-minio
spec:
  provider:
    vault:
      server: "https://vault.laughingmanlab.net"
      path: "secret"
      version: "v2"
      auth:
        jwt:
          path: "jwt-taloslmn"
          role: "r-eso_minio-jwt-taloslmn-read"
          kubernetesServiceAccountToken:
            serviceAccountRef:
              name: "external-secrets"
              namespace: "external-secrets"
            audiences:
              - "vault"
            expirationSeconds: 600

Store: vault-rpi

---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-rpi
spec:
  provider:
    vault:
      server: "https://vault.laughingmanlab.net"
      path: "secret"
      version: "v2"
      auth:
        jwt:
          path: "jwt-taloslmn"
          role: "r-eso-jwt-taloslmn-read"
          kubernetesServiceAccountToken:
            serviceAccountRef:
              name: "external-secrets"
              namespace: "external-secrets"
            audiences:
              - "vault"
            expirationSeconds: 600

Store: vault-velero

---
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-velero
spec:
  provider:
    vault:
      server: "https://vault.laughingmanlab.net"
      path: "secret"
      version: "v2"
      auth:
        jwt:
          path: "jwt-taloslmn"
          role: "r-eso_velero-jwt-taloslmn-read"
          kubernetesServiceAccountToken:
            serviceAccountRef:
              name: "external-secrets"
              namespace: "external-secrets"
            audiences:
              - "vault"
            expirationSeconds: 600

Note: I will add more secret stores as needed

Vault side: JWT auth with pinned Kubernetes SA public keys

On the Vault side, the important concept is:

• ESO creates a short-lived ServiceAccount token (TokenRequest)
• Vault validates it by checking:
  • iss (issuer) matches bound_issuer
  • aud contains the expected audience (e.g. vault)
  • sub (subject) matches the ServiceAccount identity
  • signature matches a trusted public key (JWKS → PEM pinned in Vault)

1) Get the Kubernetes issuer

kubectl get --raw /.well-known/openid-configuration | jq -r .issuer

Example output:

https://192.168.1.83:6443

2) Fetch the JWKS and convert it to PEM

kubectl get --raw /openid/v1/jwks > /tmp/taloslmn-jwks.json

Then convert JWKS → PEM (RSA public keys):

python3 - <<'PY'
import json, base64
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization

jwks = json.load(open("/tmp/taloslmn-jwks.json"))
out = []

def b64url_to_int(s):
    s += "=" * (-len(s) % 4)
    return int.from_bytes(base64.urlsafe_b64decode(s), "big")

for k in jwks["keys"]:
    if k.get("kty") != "RSA":
        continue
    n = b64url_to_int(k["n"])
    e = b64url_to_int(k["e"])
    pub = rsa.RSAPublicNumbers(e, n).public_key()
    pem = pub.public_bytes(
        encoding=serialization.Encoding.PEM,
        format=serialization.PublicFormat.SubjectPublicKeyInfo,
    ).decode()
    out.append(pem)

open("/tmp/taloslmn-sa-jwt-pubkeys.pem","w").write("".join(out))
print("Wrote /tmp/taloslmn-sa-jwt-pubkeys.pem with", len(out), "RSA keys")
PY

I keep public keys only in Git (safe to commit). If Kubernetes rotates signing keys, I regenerate this file and re-apply Terraform.

3) Terraform: enable JWT auth backend + config

resource "vault_auth_backend" "jwt_taloslmn" {
  type        = "jwt"
  path        = "jwt-taloslmn"
  description = "JWT auth for Talos LMN Kubernetes service accounts (ESO, etc.)"
  lifecycle { prevent_destroy = true }
}

resource "vault_generic_endpoint" "jwt_taloslmn_cfg" {
  depends_on           = [vault_auth_backend.jwt_taloslmn]
  path                 = "auth/${vault_auth_backend.jwt_taloslmn.path}/config"
  disable_delete       = true
  ignore_absent_fields = true

  data_json = jsonencode({
    bound_issuer = "https://192.168.1.83:6443"
    jwt_validation_pubkeys = [
      file("${path.module}/certs/taloslmn-sa-jwt-pubkeys.pem")
    ]
  })
}

4) Policies and roles

For KV v2, ESO needs read/list on:

• secret/data/... (read values)
• secret/metadata/... (list)

Example policy:

path "secret/data/k8s/taloslmn/*" {
  capabilities = ["read"]
}

path "secret/metadata/k8s/taloslmn/*" {
  capabilities = ["list"]
}

Note: Im using a naming convention and styleguide for vault which assumes one permission per policy, I will make a post about my vault styleguide some day

Example role:

• Binds sub to the ESO ServiceAccount identity:
  system:serviceaccount:external-secrets:external-secrets
• Requires aud=vault (must match the TokenRequest audience)
• Issues a token with the policy attached

{
  "backend": "jwt-taloslmn",
  "role_type": "jwt",
  "user_claim": "sub",
  "bound_audiences": ["vault"],
  "bound_claims": {
    "sub": "system:serviceaccount:external-secrets:external-secrets"
  },
  "token_policies": [
    "p-k8s_taloslmn_eso-read"
  ],
  "token_ttl": "7200"
}

End-to-end demo: Vault → ESO → Kubernetes Secret

1) Create a demo secret in Vault

vault kv put secret/k8s/taloslmn/demo-db db_password="supersecret"

2) Create a test namespace

kubectl create namespace eso-test

3) ExternalSecret manifest

This reads k8s/taloslmn/demo-db from Vault and creates demo-db-secret in Kubernetes:

---
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: demo-db-secret
  namespace: eso-test
spec:
  refreshInterval: 1m
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-rpi
  target:
    name: demo-db-secret
    creationPolicy: Owner
    deletionPolicy: Delete
  data:
    - secretKey: DB_PASSWORD
      remoteRef:
        key: k8s/taloslmn/demo-db
        property: db_password

Apply and verify:

kubectl apply -f demo-db-secret.yaml
kubectl -n eso-test get externalsecret demo-db-secret
kubectl -n eso-test get secret demo-db-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 -d; echo

Expected output:

supersecret

Cleanup:

kubectl delete namespace eso-test
vault kv delete secret/k8s/taloslmn/demo-db

Troubleshooting notes

caBundle must be of type byte

caBundle must contain literal PEM bytes (not ${VAR} placeholders). If your Vault endpoint already presents a valid trusted certificate, omit caBundle.

SecretSyncedError / auth issues

Check ESO logs:

kubectl -n external-secrets logs deploy/external-secrets --tail=200

Most common culprits:

• KV v2 policy missing secret/data/... (read) or secret/metadata/... (list)
• bound_issuer mismatch (issuer must match the token iss)
• aud mismatch (audiences: ["vault"] must match bound_audiences)
• sub mismatch (service account / namespace must match)

Validate the Vault role manually

This isolates whether the problem is Vault (auth/policy) or ESO (Kubernetes side):

TOKEN=$(kubectl -n external-secrets create token external-secrets --audience=vault --duration=10m)
vault write auth/jwt-taloslmn/login role="r-eso-jwt-taloslmn-read" jwt="$TOKEN"

Next steps

• Define a clean Vault path convention per cluster/app (e.g. secret/k8s/<cluster>/<app>/...)
• Create one ClusterSecretStore per policy boundary
• Migrate one app at a time from Sealed Secrets to ESO
• Keep private material out of Git; commit only public certificates/public keys

That’s it — Vault becomes the “source of truth” and Kubernetes just consumes secrets safely via GitOps.