Prometheus Alertmanager

Send Prometheus Alertmanager alerts to Edesy OnCall as phone calls. Add a webhook receiver with send_resolved, see how alerts map to incidents, and which labels and annotations are read out on the call.

2 min read

Point an Alertmanager webhook receiver at an Edesy OnCall alert webhook monitor, and every firing alert becomes an incident that calls your team through the monitor's escalation policy. Resolved alerts close the incident.

1. Create an alert webhook monitor#

In OnCall, go to Monitors → New, choose Alert webhook, name it (for example "Production Prometheus") and pick an escalation policy. The monitor page shows its secret URL:

https://oncall.edesy.in/ingest/<token>

2. Add the receiver#

# alertmanager.yml
route:
  receiver: edesy-oncall
receivers:
  - name: edesy-oncall
    webhook_configs:
      - url: https://oncall.edesy.in/ingest/<token>
        send_resolved: true

Keep send_resolved: true so recoveries close the incident. To page only for some alerts, add a child route that matches, for example, severity="critical" and sends to edesy-oncall, and keep your other receivers for the rest.

Reload Alertmanager after editing the file.

How alerts are mapped#

OnCall detects the Alertmanager format automatically (a JSON body with an alerts array). Each alert in the payload is handled on its own:

OnCall field Taken from
Firing or resolved The alert's status (or the payload's status if the alert has none)
Dedup key fingerprint, or else alertname, instance and job labels together
Title (read on the call) annotations.summary, else the alertname label, else "Alert"
Message annotations.description, else annotations.message, else the instance label

Titles are cut to 200 characters and messages to 300. Alerts with the same dedup key update one open incident instead of opening new ones, so Alertmanager's repeat notifications don't page again. The monitor stays down while any incident from it is open.

Write a short, clear summary annotation: it is what the on-call person hears, for example "Checkout API error rate above 5%".

Responses#

Status Meaning
202 Accepted: {"data":{"accepted":2}} (number of alerts processed)
400 invalid_payload: the body isn't JSON or has no alerts
404 Unknown ingest URL

The endpoint accepts bodies up to 1 MB and up to 120 requests a minute per IP address. To force the parser, add ?format=alertmanager to the URL.

FAQ#

Do I need an API key? No. The secret token in the URL identifies the monitor. Rotate it on the monitor page if it leaks.

Will a flapping alert call me again and again? While an incident for the same dedup key is open, repeats update it instead of opening new incidents. Once it resolves, a new firing alert opens a new incident.

Can one receiver serve several teams? Each alert webhook monitor has one escalation policy. Create one monitor per team and route alerts to each URL with Alertmanager routes.

See also: Grafana, Generic JSON and the Alertmanager phone call integration.

Was this page helpful?