The Status Page That Stayed Green

Summary: An incident degrades service for a subset of customers. The status page, driven by checks that do not represent those customers, reports normal operation. The contradiction between what users experience and what the company publishes does more damage than the outage.

Pattern

Something breaks. It does not break for everyone — it breaks for one region, one plan tier, one code path, one large customer whose usage pattern is unlike anyone else's.

The status page reports all systems operational, because the checks behind it are hitting an endpoint that is genuinely fine. Customers arriving at that page are told, in effect, that the thing happening to them is not happening.

What tends to happen

The first signal is not monitoring. It is a support ticket, then three more, then a message in a shared channel asking whether anything is going on.

Someone checks the dashboards. The dashboards look normal, because the aggregate is normal — a failure affecting four percent of requests does not move a global success-rate graph far enough to trip anything.

Meanwhile customers are checking the status page, and the status page is green. Some of them conclude the problem is on their side and start debugging their own integration. Some conclude the company is hiding an outage. Both conclusions are expensive, and the second one is worse.

The incident is eventually confirmed, usually by someone reproducing it manually. By then the question in the customer's mind has already shifted from "is this broken?" to "why did you tell me it wasn't?"

The status page is updated, sometimes after the fix has already shipped. The update describes a brief period of degraded performance. Customers who spent ninety minutes on it read that sentence with a particular expression.

Why it fails

The status page measures the wrong thing, and it measures it from the wrong place.

Most status checks are synthetic requests from infrastructure the company controls, hitting a health endpoint designed to answer quickly. That endpoint answers whether the process is running. It does not answer whether a customer in another region, authenticating with their credentials, running their query shape, against their data volume, gets a correct result in reasonable time.

The second failure is that the page is manually operated but automatically believed. Customers read it as a live instrument. Internally it is a communications artefact that someone has to decide to update — and deciding to update it during an active incident competes with actually fixing the incident, run by the same small group of people.

The third is that the threshold for posting is implicitly set by how the incident will read afterwards rather than by what customers currently need. Nobody says this out loud. It shows up as a bias toward waiting until the picture is clear, which is exactly the period during which customers most need to be told something.

Human layer

During an incident, the people who could update the status page are the people holding the diagnosis in their heads. Interrupting them to write customer communications has a real cost, and everyone in the room knows it. The natural resolution is to wait until there is something definite to say.

There is also genuine uncertainty. Early in an incident the honest statement is "some customers are seeing errors and we do not yet know the scope." That sentence feels weak to publish. It reads as though the company is not in control. So it does not get published, and the page stays green — which reads as though nothing is wrong, a claim far stronger and far less accurate than the one that felt too weak.

Afterwards, a second bias operates. The incident is over, the fix is in, and writing the update now means describing a problem customers may not have noticed. The incentive to quietly let it pass is real, and it is usually rationalised as not wanting to cause alarm.

The through-line is that every individual decision to wait was defensible, and the aggregate behaviour was to tell customers nothing while they were actively affected.

System layer

The checks are synthetic and internal. There is no measurement of what real customers actually experienced, so there is no signal that can contradict the green light.

Monitoring is aggregate. Failures concentrated in a segment — one region, one tier, one customer — are averaged into invisibility. Nothing is broken out by the dimensions along which failures actually cluster.

The status page is not wired to the alerting system. It is a separate product, updated by hand, with no coupling between an incident being declared internally and anything changing externally. Two systems that should be one.

There is no defined threshold for posting. Because the criterion is judgement rather than rule, it gets applied under time pressure by people whose attention is elsewhere, which is the condition under which judgement performs worst.

And there is no post-incident obligation to reconcile. Nothing forces a comparison between when customers were affected and when the page said so.

What it costs

The outage costs what outages cost. The green status page costs something different and more durable.

A customer who hits an error and sees an acknowledgement is dealing with a vendor having a bad day. A customer who hits an error and sees all systems operational is dealing with a vendor whose public statements do not track reality. The first is a technical problem. The second is an epistemic one, and it generalises: if the status page was wrong, what else is.

There is a direct support cost — every affected customer who believes the problem is theirs opens a ticket, and several will have spent hours before doing so. That time is charged to your relationship whether or not it appears anywhere.

There is a procurement cost that surfaces much later. Status page history is one of the few externally verifiable signals about a vendor, and a suspiciously clean one invites the question of whether it is clean or merely quiet. Sophisticated buyers ask. Unsophisticated buyers find out from someone who did.

And there is an internal cost: teams learn that the status page is not a real instrument. Once that is established, updating it becomes optional in practice, which guarantees the next occurrence.

Reduction path

Measure from where the customer sits. Checks should run from outside your infrastructure, against real request shapes, in the regions customers are actually in. A health endpoint that always answers quickly is monitoring the monitoring.

Break the signal out along the dimensions failures cluster on — region, plan, tenant, code path — so a segment-sized failure produces a segment-sized alert rather than a rounding error in the aggregate.

Couple incident declaration to external communication. Declaring an incident internally should create the external post automatically, in an investigating state, without requiring a second decision from a person who is busy. The default should be that customers are told, with silence as the deviation.

Set the threshold in advance and write it down. Something like: any incident affecting a measurable share of requests for more than a fixed number of minutes gets posted, regardless of whether the cause is understood. Removing the judgement call is the point — a rule applied badly still outperforms judgement applied under pressure.

Reconcile afterwards, as a standing part of the post-incident review: when did customers start being affected, and when did we say so. Track the gap. A number that gets looked at is a number that shrinks.

Where this gets built deliberately rather than retrofitted, it tends to look like delivery infrastructure that tracks endpoint health as a first-class signal.