Skip to main content

Pushing Metrics into ClusterNest Managed Prometheus

Every cluster comes with a write credential, scoped to exactly the push paths and nothing else

  • it can't run queries, read rules, or reach the alertmanager config API. Use it for any remote_write-speaking agent (Prometheus itself in agent mode, Grafana Agent, an OpenTelemetry Collector's Prometheus remote-write exporter).

Fetch the write password with GET /cluster/prometheus/$CLUSTER_ID/credentials, then point remote_write at the push endpoint - note this is the cluster's root URL, not /prometheus-prefixed (that prefix is for the query API only):

remote_write:
  - url: https://$PROMETHEUS_HOST/api/v1/push
    basic_auth:
      username: write
      password: $WRITE_PASSWORD

A legacy path also works, for older clients that expect it:

https://$PROMETHEUS_HOST/api/prom/push

Verifying it's working​

Push a sample series with curl (a zero-length snappy-compressed WriteRequest - snappy's block format is a length prefix followed by that many compressed bytes, so a length of zero is just the single byte \x00 - the minimal valid payload, just to confirm auth and connectivity without needing a client library). A NUL byte can't be passed as a shell argument directly (it truncates to nothing), so write it to a file first:

printf '\x00' > empty.snappy

curl -i -X POST "https://$PROMETHEUS_HOST/api/v1/push" \
  -u "write:$WRITE_PASSWORD" \
  -H "Content-Type: application/x-protobuf" \
  -H "Content-Encoding: snappy" \
  -H "X-Prometheus-Remote-Write-Version: 0.1.0" \
  --data-binary @empty.snappy

A 200 confirms the credential and endpoint are correct. For a real payload, point an actual Prometheus/Grafana Agent at the endpoint above and check its own remote_write send metrics (prometheus_remote_storage_succeeded_samples_total) rather than hand-building protobuf.

Once data has been pushed, confirm it's queryable with the readonly or admin credential (see Querying + Grafana):

curl -u "readonly:$READONLY_PASSWORD" \
  "https://$PROMETHEUS_HOST/prometheus/api/v1/query?query=up"

Retention​

How long pushed data is kept is set per cluster (retention_period - 90d by default, settable at create time or changed later, same as any other field - see the console/Terraform/ raw API tabs on Getting started for the three ways to reach a cluster's settings). Data older than the retention window is deleted, not archived - lower it only if that's the intended behavior.