Skip to main content

Alert Delivery Health

An alert channel that has never worked looks exactly like an alert channel with nothing to report. Both are quiet.

Alert Delivery Health makes that distinction visible. It reports, per channel, whether alerts are really arriving — and tells you when a channel has been accepting traffic and delivering none of it.

Why this exists​

A misconfigured webhook does not announce itself. The rule saves cleanly, the channel appears in your list, and every alert it generates disappears. You find out during an incident, when the page never arrives.

The verdict that matters most here is broken: the channel has had traffic and has never once delivered successfully. That is a configuration error that has been silently discarding alerts since the day it was created.

The four verdicts​

StatusMeaningWhat to do
healthyRecent successful delivery, no suppressionNothing
degradedSome deliveries failed, but the channel has worked beforeReview the last error — often a transient outage at the destination
suppressedThe circuit breaker is open; deliveries are being dropped right nowSee Suppression below
brokenTraffic, but not a single successful delivery, everVerify the target and credentials — this channel has never worked

Each channel also returns a plain-language remediation string, so the next step is on screen rather than in a runbook.

Suppression and circuit breakers​

When a channel fails repeatedly, Infracast opens a circuit breaker and stops attempting delivery. This protects both your alerting pipeline and the failing destination from a retry storm.

Suppression distinguishes two causes, because they need different responses:

  • Permanent — the destination rejected the delivery outright: bad credentials, a revoked token, a deleted webhook. Retrying will never succeed. Fix the channel configuration on the rule; saving the rule clears the suppression.
  • Transient — repeated but recoverable failures, such as an unreachable endpoint. Delivery retries automatically after a cooldown.

suppressed_now reports how many deliveries have been dropped since the breaker opened, so you know what you have missed.

Resetting a breaker manually​

If you have fixed the underlying problem and do not want to wait for the cooldown:

POST /api/v1/tenants/{tenantID}/alerts/rules/{ruleID}/reset-breaker

Checking delivery health​

GET /api/v1/tenants/{tenantID}/alerts/delivery-health?window_days=30

The response includes per-channel detail plus summary counters (healthy, degraded, suppressed, broken, tripped_channels) so a dashboard can badge a nav item without walking the full list.

{
  "window_days": 30,
  "healthy": 3,
  "degraded": 1,
  "suppressed": 0,
  "broken": 1,
  "tripped_channels": 0,
  "channels": [
    {
      "channel_type": "webhook",
      "status": "broken",
      "breaker_open": false,
      "remediation": "This channel has never successfully delivered an alert. Verify the target and credentials."
    }
  ]
}

The verdict survives retention​

Alert history is pruned on a retention schedule. Delivery health deliberately does not derive "has this channel ever worked?" from that history.

This matters more than it sounds. If the verdict were computed from retained rows, then once a channel's history aged out:

  • a channel that last worked months ago would masquerade as one that had never worked, and
  • a genuinely broken but low-traffic channel would disappear from the view entirely once its failure rows were pruned — exactly the channel you most need to see.

Instead, "has ever delivered" is recorded durably when it happens. Retention can prune the detail; it cannot erase the verdict.

Check delivery health before you need the alerts, not after. A channel added months ago and never exercised is the most likely one to be broken, and the least likely to be noticed — because silence from a working channel and silence from a broken one look identical until you look here.

See also​