Alert Rules & Alertmanager on ClusterNest Managed Prometheus

Every cluster runs Cortex's ruler (rule evaluation) and its own per-cluster Alertmanager instance. All of it is admin-only except the read-only rule evaluation view:

API	Purpose	Access
Rule management	Create/read/delete rule groups	admin, unprefixed /api/v1/rules/...
Rule evaluation state	Read-only view of rule groups + their current health/alert state	readonly or admin, /prometheus/api/v1/rules
Alertmanager config	Set/read/delete this tenant's Alertmanager config	admin, /api/v1/alerts
Alertmanager UI/API	Browse firing alerts, manage silences	admin, /alertmanager

Managing rule groups

A rule group is a small YAML document; groups are organized under a namespace you choose (any string):

cat > my-group.yaml <<'EOF'
name: example
rules:
  - alert: HighErrorRate
    expr: rate(http_requests_total{status="500"}[5m]) > 0.05
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Error rate above 5% for 10 minutes"
EOF

curl -X POST "https://$PROMETHEUS_HOST/api/v1/rules/my-namespace" \
  -u "admin:$ADMIN_PASSWORD" \
  -H "Content-Type: application/yaml" \
  --data-binary @my-group.yaml

List and delete:

curl -u "admin:$ADMIN_PASSWORD" "https://$PROMETHEUS_HOST/api/v1/rules"
curl -X DELETE -u "admin:$ADMIN_PASSWORD" \
  "https://$PROMETHEUS_HOST/api/v1/rules/my-namespace/example"

GET /api/v1/rules with no groups configured yet returns 404, not an empty list - that's Cortex's own "nothing here" response for the ruler, not an error.

Rule evaluation state (read-only)

The same rule groups, plus their live evaluation state (health, last eval time, current alert/value) - reachable by readonly too, since it's under the /prometheus prefix:

curl -u "readonly:$READONLY_PASSWORD" "https://$PROMETHEUS_HOST/prometheus/api/v1/rules"

Alertmanager config

Set where alerts route to (receivers, routing tree). The request body is Cortex's own wrapper around a standard Alertmanager config, not the Alertmanager config directly - the actual config goes in alertmanager_config as an embedded YAML string:

cat > alertmanager-config.yaml <<'EOF'
template_files: {}
alertmanager_config: |
  route:
    receiver: default
  receivers:
    - name: default
      webhook_configs:
        - url: https://hooks.example.com/alerts
EOF

curl -X POST "https://$PROMETHEUS_HOST/api/v1/alerts" \
  -u "admin:$ADMIN_PASSWORD" \
  -H "Content-Type: application/yaml" \
  --data-binary @alertmanager-config.yaml

Read it back the same way (GET instead of POST) - the response has the identical wrapper shape. This config can carry secrets (webhook URLs, PagerDuty integration keys), which is why this endpoint is admin-only rather than open to readonly.

Browsing alerts and managing silences

The Alertmanager's own native UI and API (distinct from the config endpoint above) is at /alertmanager:

curl -u "admin:$ADMIN_PASSWORD" "https://$PROMETHEUS_HOST/alertmanager/api/v2/status"

Or open https://$PROMETHEUS_HOST/alertmanager in a browser (basic-auth prompt, same admin credential) for the native silence-management UI.