Skip to main content

Alert Delivery Health

An alert channel that has never worked looks exactly like an alert channel with nothing to report. Both are quiet.

Alert Delivery Health makes that distinction visible. It reports, per channel, whether alerts are really arriving — and tells you when a channel has been accepting traffic and delivering none of it.

Why this exists​

A misconfigured webhook does not announce itself. The rule saves cleanly, the channel appears in your list, and every alert it generates disappears. You find out during an incident, when the page never arrives.

The verdict that matters most here is broken: the channel has had traffic and has never once delivered successfully. That is a configuration error that has been silently discarding alerts since the day it was created.

The four verdicts​

StatusMeaningWhat to do
healthyRecent successful delivery, no suppressionNothing
degradedSome deliveries failed, but the channel has worked beforeReview the last error — often a transient outage at the destination
suppressedThe circuit breaker is open; deliveries are being dropped right nowSee Suppression below
brokenTraffic, but not a single successful delivery, everVerify the target and credentials — this channel has never worked

Each channel also returns a plain-language remediation string, so the next step is on screen rather than in a runbook.

Suppression and circuit breakers​

When a channel fails repeatedly, Infracast opens a circuit breaker and stops attempting delivery. This protects both your alerting pipeline and the failing destination from a retry storm.

Suppression distinguishes two causes, because they need different responses:

  • Permanent — the destination rejected the delivery outright: bad credentials, a revoked token, a deleted webhook. Retrying will never succeed. Fix the channel configuration on the rule; saving the rule clears the suppression.
  • Transient — repeated but recoverable failures, such as an unreachable endpoint. Delivery retries automatically after a cooldown.

suppressed_now reports how many deliveries have been dropped since the breaker opened, so you know what you have missed.

Resetting a breaker manually​

If you have fixed the underlying problem and do not want to wait for the cooldown:

POST /api/v1/tenants/{tenantID}/alerts/rules/{ruleID}/reset-breaker

Checking delivery health​

GET /api/v1/tenants/{tenantID}/alerts/delivery-health?window_days=30

The response includes per-channel detail plus summary counters (healthy, degraded, suppressed, broken, tripped_channels) so a dashboard can badge a nav item without walking the full list.

{
  "window_days": 30,
  "healthy": 3,
  "degraded": 1,
  "suppressed": 0,
  "broken": 1,
  "tripped_channels": 0,
  "channels": [
    {
      "channel_type": "webhook",
      "status": "broken",
      "breaker_open": false,
      "remediation": "This channel has never successfully delivered an alert. Verify the target and credentials."
    }
  ]
}

The verdict survives retention​

Alert history is pruned on a retention schedule. Delivery health deliberately does not derive "has this channel ever worked?" from that history.

This matters more than it sounds. If the verdict were computed from retained rows, then once a channel's history aged out:

  • a channel that last worked months ago would masquerade as one that had never worked, and
  • a genuinely broken but low-traffic channel would disappear from the view entirely once its failure rows were pruned — exactly the channel you most need to see.

Instead, "has ever delivered" is recorded durably when it happens. Retention can prune the detail; it cannot erase the verdict.

Check delivery health before you need the alerts, not after. A channel added months ago and never exercised is the most likely one to be broken, and the least likely to be noticed — because silence from a working channel and silence from a broken one look identical until you look here.

See also​

Large scans send one summary, not hundreds of alerts​

When a single scan produces more than 10 findings that match a rule, each of the rule's channels receives one summary instead of one alert per finding. The summary shows the count by severity and by framework, lists the most severe findings, and links to the full list. Resolved findings are always reported as a summary.

The first scan of a workspace is labelled Initial baseline: every finding is new only because nothing existed before. Later scans alert only on changes — findings that are new, or that were resolved and have come back.

Finding alerts cover only the frameworks enabled in the rule creator's settings — the same frameworks shown on their Dashboard. If no frameworks are selected, all are included.

Other alert types (for example Infrastructure appeared) are limited to 20 per rule and channel every 10 minutes. Anything beyond that is recorded once in alert history as throttled.

Scheduled scans that keep failing​

When a scheduled discovery scan fails 3 times in a row, Infracast raises a Scan Failed alert with the error message, how many times it has failed, when it last succeeded and when it will retry next. If the scan keeps failing, a reminder follows every 5 further failures, so a broken schedule cannot go quiet. When the scan works again, a Scan Recovered alert follows.

A single failure is not alerted: throttling and brief network errors usually clear on the next retry.

To receive these, create an alert rule for Scan Failed (and optionally Scan Recovered) under Alerts. The Connectors → Jobs schedule list also shows each schedule's status, and marks one that is failing with the number of failures in a row.

The default "Scheduled scan failures" rule​

Every tenant starts with an alert rule named Scheduled scan failures. It emails all of the tenant's active admins when a scheduled scan fails 3 times in a row (with reminders while it keeps failing) and when it recovers. Admins added later are included automatically.

You can edit it — for example to send to a Slack channel instead — or delete it. A deleted default rule is not re-created.

When creating your own email rule, you can enter tenant-admins as the address to send to every active tenant admin.