Skip to main content

Querying ClusterNest Managed Prometheus and Wiring Up Grafana

PromQL over HTTP​

The query API lives under the /prometheus prefix on the cluster's endpoint, using the readonly (or admin) credential:

curl -u "readonly:$READONLY_PASSWORD" \
"https://$PROMETHEUS_HOST/prometheus/api/v1/query_range?query=up&start=$START&end=$END&step=15s"

A legacy path also works: https://$PROMETHEUS_HOST/api/prom/api/v1/query.

readonly is scoped to query paths only - it deliberately can't reach the alertmanager config-CRUD API (/api/v1/alerts), since that config can carry receiver secrets (webhook URLs, PagerDuty keys). If you need that, use the admin credential instead.

Grafana datasource​

Add a Prometheus datasource in Grafana:

FieldValue
URLhttps://$PROMETHEUS_HOST/prometheus
AuthBasic auth
Userreadonly
Passwordthe readonly password from credentials

The /prometheus path segment in the URL is required - Grafana's Prometheus datasource issues requests like GET/POST /prometheus/api/v1/query_range and GET/POST /prometheus/api/v1/rules against whatever base URL you configure; without the prefix, every request 404s against Cortex's actual route table.

Grafana's datasource also issues POST (not just GET) for some queries - the readonly credential already allows both, so this works without extra configuration.

Alert rules in Grafana​

Grafana's Alerting UI, when pointed at this datasource, reads existing rule evaluation state (not rule management) from /prometheus/api/v1/rules - covered by readonly's access. Actually creating or editing rule groups is a separate, more restricted API - see Alert rules & Alertmanager.