JWT/OIDC Federation for ClusterNest Managed Prometheus

Alongside the static admin/readonly/write credentials (which always exist and can't be turned off), a cluster can also trust one or more external JWT issuers directly - your own OIDC identity provider, or a Kubernetes cluster's own apiserver authenticating its workloads via their projected ServiceAccount tokens. A token presented as a Bearer credential is checked against every configured source; if it verifies, it receives whichever access level(s) its claims match.

One precedence rule worth knowing: if a request's Authorization header carries a Basic credential (as curl -u sends), it's judged on that alone - a misconfigured Basic credential rejects the request even if a valid JWT is also on offer somewhere else. Don't mix the two on the same request when testing.

Configuring a source

Each entry in prometheus_jwt_sources:

Field	Meaning
name	Identifies this source in error logs; unique per cluster.
issuer	The trusted iss claim.
jwks_url or oidc_discovery_url	Where to fetch signing keys - exactly one of the two.
audiences	Allowed aud values; empty means any.
ca_cert	Base64 PEM CA bundle, if the source's TLS isn't publicly trusted.
grants	List of {role, claims} - see below.

A grant's role is admin/readonly/write, same meaning as the static credentials. Its claims is an exact-match map (e.g. {"sub": "..."}) a token must satisfy to receive that grant; an empty map matches any token the source verifies. A token can match more than one grant - its effective access is the union (a token matching both a write grant and a broader, claims-free admin grant gets admin access).

Console

Open the cluster's settings and add an entry under JWT Authentication - issuer, jwks_url or oidc_discovery_url, optional audiences, and one or more role/claims grants, matching the fields above.

Terraform

resource "clusternest_prometheus" "metrics" {
  name            = "metrics"
  tier            = "standard"
  organization_id = 123

  prometheus_jwt_sources = [
    {
      name     = "idp"
      issuer   = "https://idp.example.com"
      jwks_url = "https://idp.example.com/jwks"
      grants = [
        { role = "write", claims = { sub = "remote-write-agent" } },
      ]
    },
  ]
}

See the resource reference for the full schema.

Raw API

PUT replaces the whole cluster, not just the fields you send

PUT /cluster/prometheus/$CLUSTER_ID is a full replacement, not a merge - any field you leave out of the body resets to its schema default, not its current value. GET the cluster first, merge the field(s) you're changing into that full response, and PUT the merged result back - never a body containing only prometheus_jwt_sources. (Terraform doesn't have this problem - it always sends the complete desired state on every apply.)

curl "https://api.clusternest.com/cluster/prometheus/$CLUSTER_ID" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  | jq '.prometheus_jwt_sources += [{
      "name": "idp",
      "issuer": "https://idp.example.com",
      "jwks_url": "https://idp.example.com/jwks",
      "grants": [{"role": "write", "claims": {"sub": "remote-write-agent"}}]
    }]' > cluster.json

curl -X PUT "https://api.clusternest.com/cluster/prometheus/$CLUSTER_ID" \
  -H "Authorization: Bearer $ACCESS_TOKEN" \
  -H "Content-Type: application/json" \
  --data-binary @cluster.json

(+= appends to the existing array rather than replacing it, so any JWT sources already configured stay in place.)

Takes effect within seconds, no cluster restart.

Worked example: Kubernetes ServiceAccount tokens (EKS)

A workload running inside a Kubernetes cluster can push metrics using its own ServiceAccount token as a Bearer credential instead of a static password - no secret to distribute or rotate, since kubelet mints and refreshes the token automatically. This is the same mechanism ClusterNest's own internal metrics pipeline uses to let one cluster push into another.

Any cluster with ServiceAccountIssuerDiscovery enabled works this way; EKS has it on by default, so this example uses EKS - the same steps apply to any conformant Kubernetes cluster, just with a different issuer URL.

1. Find your EKS cluster's OIDC issuer

The same issuer URL IRSA already uses:

aws eks describe-cluster --name my-cluster --query "cluster.identity.oidc.issuer" --output text
# https://oidc.eks.<region>.amazonaws.com/id/<cluster-id>

2. Configure the JWT source

{
  "name": "eks",
  "issuer": "https://oidc.eks.<region>.amazonaws.com/id/<cluster-id>",
  "oidc_discovery_url": "https://oidc.eks.<region>.amazonaws.com/id/<cluster-id>/.well-known/openid-configuration",
  "audiences": ["cortex"],
  "grants": [
    {
      "role": "write",
      "claims": { "sub": "system:serviceaccount:monitoring:prometheus-agent" }
    }
  ]
}

The sub claim on a Kubernetes-issued ServiceAccount token always has the form system:serviceaccount:<namespace>:<service-account-name> - match on it directly instead of trying to scope by anything else.

3. Project the token into the pod

Give the pod a token bound to the cortex audience above (matching audiences), refreshed automatically by kubelet:

spec:
  serviceAccountName: prometheus-agent
  containers:
    - name: prometheus-agent
      volumeMounts:
        - name: cortex-token
          mountPath: /var/run/secrets/tokens
  volumes:
    - name: cortex-token
      projected:
        sources:
          - serviceAccountToken:
              path: cortex-token
              audience: cortex
              expirationSeconds: 3600

4. Point remote_write at the token file

remote_write:
  - url: https://$PROMETHEUS_HOST/api/v1/push
    authorization:
      credentials_file: /var/run/secrets/tokens/cortex-token

Prometheus/Grafana Agent re-reads credentials_file on every request, so kubelet's periodic token refresh (every expirationSeconds, well before actual expiry) is picked up without a pod restart.

Testing a source

Decode whatever token you're using (jwt.io, or kubectl create token <sa> --audience=cortex for a quick manual one) to confirm its iss/aud/sub match what's configured, then push a zero-timeseries request (see Pushing metrics in for why empty.snappy has to be a file, not an inline argument):

printf '\x00' > empty.snappy

curl -i -X POST "https://$PROMETHEUS_HOST/api/v1/push" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/x-protobuf" \
  -H "Content-Encoding: snappy" \
  --data-binary @empty.snappy

A 401 means the token didn't verify (wrong issuer/audience/signature); a 403 means it verified but no grant's claims matched; 200 means it worked.