The worst thing a monitoring tool can do is lie to you at 3 a.m. After the second false alarm you start ignoring alerts, and then the real outage slips by. So the central design decision in Tenium’s checker is simple: one region cannot open an incident. A quorum of independent regions can. Today that quorum is 2 of our 2 live regions.
This post explains how that works, what it costs in alert latency, and what you get in the incident record when we do page you.
The setup
Tenium runs one checker process per region. 2 regions are live: Amsterdam and London. Each process is a plain Node worker that talks to the same Postgres database. There is no message queue and no Redis; the database is the coordination layer.
Every monitor has an interval (30 seconds on Pro) and a set of regions. The scheduler in each region claims monitors that are due with a single query:
UPDATE monitors SET next_check_at = now() + interval
WHERE id IN (
SELECT id FROM monitors
WHERE next_check_at <= now() AND region = $1
ORDER BY next_check_at
LIMIT 50
FOR UPDATE SKIP LOCKED
) RETURNING *;FOR UPDATE SKIP LOCKED means several processes can run the same query at the same time and never pick the same row. It is boring and it has not failed us.
A normal check
A check is an HTTP request with a 10-second timeout, a fixed User-Agent: Tenium-Monitor/1.0, any custom headers you configured, and timing captured at each phase: DNS, TCP connect, TLS handshake, time to first byte, download. We record the status code, response size, the resolved IP, certificate expiry for https, and the first 2 KB of the body if the check failed.
The result goes into a checks table with the region code. That is where the uptime percentages and the response-time charts come from.
When a check fails
Say Frankfurt gets a 503 from shop.clienta.com. Here is what happens, in order:
- The failure is recorded as a check result, but no incident is opened and nobody is alerted. The monitor’s state in that region becomes “suspect”.
- Other regions are asked to check immediately. The scheduler inserts rows into a
confirmationstable, one for each region asked, picked among the regions that are alive. Those regions pick up the rows within one second, because confirmations are claimed ahead of scheduled checks. - Both run the exact same request, same headers, same timeout.
- If the quorum agrees it is down, the monitor transitions to
down, an incident is opened, and alerts are dispatched. The quorum is three regions, or fewer if fewer are available to the monitor. That means the original failure plus the confirmations. The rule is the same whatever the region count, so a bigger network makes the evidence broader without changing the logic. - If the confirmations succeed, the monitor stays
up, the original failure is kept in history as a regional blip, and you see a small grey marker on the chart instead of a red one.
The whole sequence takes 5–15 seconds on top of the original check. That is the price of not lying to you.
A word on honesty about the design. With the quorum at three, the confirmation is strict: a failure that fewer regions than the quorum can see does not open an incident, although those failures stay in the history. We think that is the right default for avoiding false alarms, and it is why we widen the network rather than loosen the rule.
What counts as “down”
A check fails when any of these is true:
- Connection refused, DNS failure, or timeout
- Status code outside the expected set (default 2xx and 3xx)
- The keyword is missing (or present, for “must not contain”)
- A JSON assertion fails
- The TLS handshake fails, or the certificate is expired or does not match the hostname
A slow response is not down. Response-time alerts are a separate rule based on the p95 of the last ten checks, so one slow request from a single region does not trigger anything.
The incident record
When we do alert you, the message and the incident page contain:
- Which regions confirmed, with each one’s error and timing
- The HTTP status and the first lines of the response body, so “503 from Cloudflare” and “503 from nginx” look different
- The IP the hostname resolved to in each region, which catches DNS problems instantly
- The certificate expiry if relevant
- A link to acknowledge, which stops escalation
Recovery is confirmed the same way: the monitor must succeed from the quorum of regions before we close the incident and send the “back up, down for 2 m 10 s” message. Flapping sites therefore produce one incident, not twenty.
The trade-offs
Confirmation adds up to 15 seconds before an alert. We think that is right for websites; a 15-second delay has never cost anyone a client, and a false 3 a.m. alert has.
Confirmation also means regional outages are reported as regional. If your site is unreachable only from New York because of a peering problem, you will see it on the chart and the status page can show it as degraded for that region, but we will not call it down unless the other regions agree. If you need a regional alert, you can set a monitor to use a single region, and it will alert on that region alone, with the trade-off clearly labelled.
Why not just retry from the same region?
Because a retry from the same place tests the same route, the same resolver cache and the same peering. Most false alerts come from exactly those. A second region is a genuinely independent observer.
The source IPs for every live region are listed at /checker-ips with a JSON feed, so you can allow-list them in firewalls and WAFs. If you run Cloudflare, read the WAF allow-list guide.
Related
The user-facing description is in regions and confirmation. To see where confirmation matters most, read about multi-location checks, website monitoring and API monitoring, or run the is-it-down tool against your own site. If you are choosing a vendor on this basis, see how UptimeRobot and Better Stack compare.
Related
Related Tenium pages
Monitor your first site free
30 monitors free, checks from 2 regions, no card. SSL and domain expiry from $7 a month.