Incidents with evidence, acknowledgement and notes

The life of one incident

An incident starts when a failure has been confirmed, not when the first check fails. At that moment the incident records a cause (a status code, a timeout, a certificate error, a missed heartbeat), the regions that confirmed it, the first lines of the response, the resolved IP and, when we can tell, the upstream provider involved. Contacts are alerted, the status page flips, and a timeline begins. It ends when recovery is confirmed the same way, and the recovery alert states how long the outage lasted.

Severity is not binary. A site that answers but serves a bot challenge is recorded as blocked rather than down, because the right fix is different. A response that is slower than your threshold, or a page whose content changed under a change-detection monitor, opens a degraded incident. If a degraded incident escalates to down, the first is closed silently and the second opened, so the timeline stays one clean sequence.

Acknowledging, from wherever you are

Acknowledgement tells the system a human has taken it. You can do it from the incident page, from the link in an email alert, from the API, or in Telegram by replying /ack. The Telegram command acknowledges the most recent open incident for the account that the chat is linked to, which is convenient on a quiet night and worth remembering when two things are broken at once.

Acknowledging stops further escalation steps. It does not close the incident and it does not silence the recovery notification, so the people who were paged still hear when the site is back. The audit log records who acknowledged and through which route.

Evidence that holds up in a client call

The point of capturing the response excerpt and IP is that arguments about what happened end quickly. A 503 page from the host’s load balancer looks different from a PHP fatal error, and a resolved IP that differs between regions shows a DNS or CDN issue. The incident view lists this per region, and the API returns the same fields, including confirmedBy and an evidence array with status code, time, IP and body excerpt. An example of the shape: a 130-second incident confirmed by several regions, each with a 503 and a 412 ms response, acknowledged 50 seconds after it opened.

Notes and the customer-facing story

Notes are for the human part: “host restarted PHP-FPM”, “client uploaded 40 MB images”. They appear in the incident timeline and in the monthly report, which turns a technical event into something you can explain. You can add them in the app or with a POST to the incident notes endpoint, so an automated runbook can leave a trace of what it did.

For what the public sees, incidents opened by monitors show on the status page on their own, and you can post manual incidents with updates for problems the monitors do not capture, such as a payment provider outage that does not break your own URLs. Those updates go to email subscribers and the RSS feed. See status pages for setup.

A ten-minute weekly review

Incidents become useful when you look at them in bulk. Once a week, open the list and check three things: which monitors opened the most incidents, which incidents were never acknowledged, and which ones have no note. Monitors that keep reappearing usually point at a real weakness such as an undersized host or a flaky cron job. Unacknowledged incidents that recovered by themselves are often alerts going to the wrong contact. An empty notes column is a hint to write two lines before the monthly report goes out, while you still remember what the fix was.

Where this is intentionally thin

This is incident handling for a small team or an agency, not an incident-response platform. There are no on-call calendars with rotations, no runbook automation of our own, and no postmortem templates. What there is: per-contact delays, a second Telegram or email contact on Pro, and PagerDuty on Agency for teams that already live there. The escalation mechanics are covered on alert escalation. A flapping site is deliberately folded into one incident with several state changes so that a bad five minutes is one page, not forty. Hourly alert caps per plan also protect the worker and your inbox if something truly pathological happens.