Why "99.9% Uptime" in a Contract Guarantees Nothing
Plenty of contracts state "99.9% monthly uptime guaranteed." Yet that single sentence decides nothing when an actual outage hits, because it never defines what counts as "up," who measures it, or whose measurement wins when the two sides disagree.
A number without a measurement definition produces disputes, not reliability. 99.9% allows roughly 43 minutes of downtime per 30-day month. But whether you count those 43 minutes as "time the server process was dead" or "time users could not complete an order" changes the result by more than tenfold. It is not unusual for a health check endpoint to keep returning 200 while the payment gateway integration is severed and revenue sits at zero for three hours. The server was "healthy," so contractual uptime records 100%.
The reverse happens too. The infrastructure team logs a five-minute restart during a night deploy as an outage, but only two real users were active and both succeeded on retry. From a user experience standpoint, those five minutes were effectively harmless. This is why the measurement subject must be the user journey, not the server.
One more factor compounds all of this: projects handed over after development finishes without operational ownership ever being assigned. When something breaks, the client calls the vendor, the vendor replies "that is outside contract scope," and the service stays down in between. Operational structure must be designed at project kickoff, not at delivery.
The Actual Order for Defining SLIs and SLOs
Step 1: Pick three to five critical paths
Trying to measure every API means managing none of them. Select only the user journeys tied directly to revenue. For commerce that is login → product browse → cart → order → payment. For a B2B system it might be login, data query, report generation, and batch processing. Past five paths, management cost exceeds the benefit.
Step 2: Choose the right indicator per path
Different paths call for different indicators.
Decide the measurement point as well. Server logs and client-side measurements produce different numbers. To represent the user's view, measuring at or above the load balancer is the safer choice.
Step 3: Do not set the target at 100%
A 100% target makes every change a liability and deployments grind to a halt. A 99.9% target creates a 43-minute monthly error budget. While budget remains, you ship new features; once it is spent, you stop deploying and invest in stabilization. This is where the target becomes a practical lever on release velocity.
Targets should differ by path. Payment at 99.95%, product browse at 99.5%, and report generation at 99% is a realistic split. Applying one number everywhere lets the most expensive requirement set the cost for the entire system.
Step 4: Separate SLA from SLO
The SLA is a contract; the SLO is an operating standard. Setting them to the same number means the moment you miss your SLO you are also in breach. Keep the internal SLO one notch stricter than the contractual SLA — if the SLA is 99.5%, set the SLO at 99.9%. That buffer buys you time to respond before a breach occurs.
Who Responds at 3 A.M., and How
The reality of on-call rotation
Textbook on-call means at least six people rotating weekly, so each person is on duty once every six weeks. Rotating three or fewer people guarantees burnout.
Realistic alternatives for understaffed organizations look like this.
Alert design: what wakes you versus what waits for morning
The most common cause of on-call collapse is not headcount — it is alert fatigue. If 50 alerts arrive daily and only two require action, people soon ignore all of them.
Pages must be written on symptoms, not causes. "Payment success rate below 90% for five minutes" is a good page condition; "CPU above 80%" is not. The former is always a problem; the latter may be harmless.
Severity levels and role separation
| Level | Definition | Response | Customer notice |
|-------|-----------|----------|-----------------|
| S1 | Full service outage, data loss | Page immediately, all hands | Within 30 min |
| S2 | Partial failure of a core function | Page immediately | Within 1 hour |
| S3 | Secondary feature down, workaround exists | Business hours | On request |
| S4 | Degraded performance, minimal user impact | Add to backlog | Not required |
For S1 and S2, separate the roles without exception. When the person fixing the outage is also the person communicating about it, both jobs fail. Split incident command, technical remediation, and customer communication across at least two people.
What makes a runbook actually get used
Most runbooks are never opened after being written. The ones that do get used share these traits.
Becoming an Organization That Does Not Repeat the Same Outage
Blameless postmortems
The moment a person is named as the cause, the organization starts hiding information. Ask not "why did you do that" but "what made that decision look reasonable at the time." If an engineer was able to run a destructive command, that is a system failing to prevent it, not individual carelessness.
Every postmortem document must include:
Write one for every S1 without exception, and for S2 when it recurs. Within five business days of the incident is a realistic standard.
Keeping prevention items from disappearing
Postmortems usually fail at the last step: items go into the backlog, get outranked by feature work, and are still sitting there six months later.
Metrics more useful than MTTR
MTTR is statistically meaningless at low incident counts and is easily skewed by a single large outage. These metrics tell you more.
Options for Organizations Without Internal Capacity
| Model | Cost sense | Scope of responsibility | Best fit |
|-------|-----------|------------------------|----------|
| In-house | Salary for 2–3 engineers | Full | Service is core to the business and hiring is feasible |
| MSP | Scales with infrastructure | Mostly infrastructure layer | Delegating servers and network only, handling the app internally |
| Maintenance contract | 10–15% of build cost per year | Mostly bug fixes | Stable systems with few changes |
| Subscription operations | Flat monthly | Monitoring, incident response, improvement | Ongoing changes, but too early for dedicated staff |
An MSP will watch whether servers are alive, but usually cannot detect a drop in payment success rate. Make it explicit at contract time who owns application-level SLOs.
Deliverables you must receive at handover
When taking over operations from a development vendor, handover is not complete without these.
Rollback procedures and certificate expiry dates are the two most commonly missing items. Full outages caused by expired certificates remain one of the most reliably recurring incident types year after year.
Build Your Operations Practice With POLYGLOTSOFT
POLYGLOTSOFT designs operations alongside development from project kickoff. Critical-path SLI/SLO definitions, symptom-based alerting, severity-tiered runbooks, and a postmortem process are all included in project deliverables.
Once development completes, our SM plans take over operations. Basic SM ($100/mo) covers bug fixes and server monitoring, Standard SM ($100/mo) adds incident response and recovery, and Pro SM ($300/mo) provides performance monitoring and database tuning with a dedicated PM. Team SM ($500/mo) extends to infrastructure scaling and on-site response.
You can also start with a reliability assessment of a system already in production. Contact [POLYGLOTSOFT](https://polyglotsoft.dev) and we will review your current monitoring setup and incident response procedures, then propose an improvement roadmap.