Cloud Native Platforms: Run

Audience: SREs (Site Reliability Engineers), platform engineers, engineering managers running production systems

Reading time: 8 minutes

Series: Cloud Native Platforms. Build, Run, Evolve. This is Part 2 of 3.

---

Most systems are designed thoughtfully.

Most operations are inherited reactively.

The systems that survive are not the ones built with the most care. They are the ones operated with the most discipline. Production has a way of revealing every shortcut taken during design and every assumption left unverified.

This post is about what it takes to operate a platform once the build is done.

How they are run, not how they are built

Systems are not defined by how they are built. They are defined by how they are run.

A well-designed system that is operated reactively will fail in production. A modestly designed system that is operated with discipline will outperform it. Five operational disciplines decide which side of that line a platform lives on. Each one is engineering work, not a checklist for someone else to handle.

Figure 1. The incident lifecycle as a state machine. The states are not optional steps. They are the contract between the team and the system.

1. Observability is the backbone of reliability

Without observability, every operation becomes a guess. As systems grow, the cost of guessing rises faster than the cost of seeing.

Part 1 of this series argued that observability is a design property: instrumentation contracts, request id propagation, structured logging schemas. Production is where those design choices either pay off or do not. Strong observability in production is a contract that lets any engineer answer three questions in minutes: what failed, why it failed, and what the impact was. The shape of that contract matters more than the tool that implements it. (This three-question framing is community-popularised through the SRE community and writers such as Charity Majors. See Honeycomb's What is Observability for the canonical articulation of the three-pillars and question framing; the substance is older than the framing.)

• Dashboards organised around user journeys, not infrastructure components
• Service level indicators (SLIs: the specific measurements you care about, e.g., success rate, p99 latency) chosen from the user's perspective, not the database's
• Alerts that page only on burn-rate against an SLO (Service Level Objective: the target value of an SLI, e.g., 99.9% of requests complete in under 800ms over a rolling month) using a multi-window strategy. A short window catches fast burns; a long window catches slow drifts. This is what makes SLOs operational rather than decorative.
• Sampling and retention tuned for cost, but never for blind spots
• The distinction between MTTA (mean time to acknowledge: how fast someone notices) and MTTR (mean time to restore: how fast service returns) tracked separately. Conflating them hides whether the team's bottleneck is detection, response, or fix.

In practice

The pattern that works: rebuild the operational view around two or three user journeys (sign-in, place order, view history) rather than per-component charts. Tie alerts to error budget burn rather than raw threshold crossings. Track MTTA and MTTR separately so the team's actual bottleneck (detection, response, or fix) is visible. The investment is rethinking what to measure, not buying a new tool. The return is that incidents stop being discovered by customer complaints first. Teams that make this shift typically find their existing telemetry was sufficient; only the questions being asked of it were wrong.

If a dashboard cannot answer "what is the user experiencing right now", it is not an observability dashboard. It is decoration.

2. Alerts are signals, not notifications

More alerts do not mean better monitoring. In practice, the opposite is true. Once alerts outpace the team's ability to act, important signals start getting missed.

Effective alerting works to a small set of rules:

• Severity that maps to action, not to technical category
• Ownership baked in, never inferred at runtime
• Thresholds tied to user impact, not raw metric values
• Noise treated as a defect, with a regular review cadence
• Suppression and grouping for known multi-alert patterns

In practice

The pattern that works: audit every alert against one test, "what action would I take in the next five minutes if this fires now?" Demote alerts with no answer to dashboards. Remove alerts where the answer is the same as another alert's. Group related alerts so one incident produces one page, not twelve. Most teams discover their alert volume drops by an order of magnitude after a thorough audit, and the alerts that remain start getting trusted again. Trust is the precondition for every other operational practice. Without it, on-call rotations decay into noise filtering and the real signals get missed.

Figure 2. From raw events to pages, in approximate orders of magnitude. The numbers vary by team and workload; what does not vary is that each stage needs to remove one to two orders of magnitude of noise. Teams that page on raw events end up with on-call rotations nobody trusts.

3. Incident response is a practiced muscle

Failures are inevitable. Unstructured response is not.

The teams that recover quickly do not improvise during incidents. They follow a structure that has been practiced when nothing was on fire. The structure is intentionally simple, because incident time is the worst time to negotiate roles.

• Clear roles: incident lead, communications lead, scribe, subject matter expert (the RACI model, Responsible-Accountable-Consulted-Informed, adapted for incident response)
• Defined escalation paths with clear handoff criteria. Escalation means re-paging to a higher tier or specialist, not returning to detection. The lifecycle diagram in Figure 1 makes the distinction explicit.
• Runbooks for the top failure modes, kept short enough to actually be read
• Status communication on a fixed cadence, even when there is nothing new to say. Customer comms and internal comms are tracked separately.
• Blameless postmortems (focus on the system that allowed the failure, not the person who pushed the button) that produce action items the team actually completes
• Game days: scheduled exercises that simulate failure modes (region outage, dependency unavailability, traffic spike) under controlled conditions, so gaps in runbooks are found before incidents do

In practice

The pattern that works: name the incident lead and the comms lead before the first message goes out. Write runbooks short enough to be scannable at 3 AM. Run blameless postmortems with action items that actually get tracked to completion. Schedule game days quarterly so the runbooks are exercised before real incidents. Teams that operate with this structure do not have more engineers; they have engineers who are not single points of failure during recovery. The deepest experts stay the deepest experts, but the platform stops depending on whether they happen to be online.

Implementation note

A short, well-structured runbook outperforms a long, exhaustive one. The goal during an incident is not to think. It is to act on a procedure that has been thought through in calmer times.

# Runbook header pattern (keep it scannable in incident time)
title: High latency on order API
slo_protected:                  # this runbook protects two SLOs
  - order-completion-success
  - order-completion-latency
severity:                       # derived from burn rate, not declared
  fast_burn: P1                 # 14.4x budget burn over 1 hour => page now
  slow_burn: P2                 # 6x budget burn over 6 hours => investigate
owner: payments-team
indicators:                     # triggers for evaluation, not severity
  - p99 (99th-percentile) latency exceeds the SLO target for 5 min
  - error rate exceeds the SLO target for 3 min on order-completion
first_actions:
  - Open the order-journey dashboard. Confirm impact in business terms.
  - Check Service Bus queue depth and dead-letter rate (the most common
    cause of API latency under load is downstream backpressure)
  - Verify Cosmos DB RU/s saturation and partition hotspots
  - Inspect the most recent deployment for behavioural changes
escalate_if:
  - Latency does not recover in 15 min
  - Error rate exceeds 5% (fast burn against the SLO)
  - Customer reports arrive before our own signals do
rollback_path:
  - Feature flag "new-order-pipeline" can be disabled per-tenant
  - Last known good deployment id is in the release tracker
note_on_scaling:
  # CPU is rarely the cause of latency in this service. Scale only after
  # confirming the bottleneck is compute, not a downstream dependency or
  # queue depth. Adding capacity to a saturated downstream amplifies the
  # incident; it does not resolve it.

The general principle behind that last note travels beyond this runbook: scale-out is the right remediation for compute saturation, not for downstream saturation. When latency rises because a database, queue, or external dependency is saturated, adding capacity in front of the bottleneck moves more requests into the bottleneck and makes the incident worse. This is one of the most common operational mistakes when the dashboard shows red and the on-call instinct says "add more".

4. Release confidence is engineered

Releases get harder as systems grow. The platforms that ship confidently at scale have engineered the path, not learned to fear it.

The patterns that change the math:

• Feature flags that allow change without deploy
• Canary deployments (releasing the new version to a small slice of traffic first, watching error budget burn before continuing) that surface problems on a small slice
• Gradual rollouts with automated rollback triggers
• Database migrations split from application releases
• Release coordination that scales with services, not with team size

In practice

The pattern that works: every change ships behind a feature flag, canary deployments take a small slice of traffic first, and rollback is a one-click step in the pipeline rather than a procedure to be invented during an incident. The cost is the discipline of building rollback paths and exercising them. The return is releases that stop being events. Issues that previously triggered full rollbacks get isolated to a slice and rolled back automatically before they reach most users. The willingness to ship smaller, more frequent changes follows directly from the confidence that bad changes can be undone fast.

Big releases feel safe because they are rare. They are actually risky because every change rides together.

5. Reliability is continuous, not a milestone

Reliability is not achieved through tools alone. It requires continuous refinement, feedback-driven improvement, and a budget that the team can spend on operational work without negotiating each time.

The disciplines that keep systems reliable over years are codified well in the SRE-book framing of service level objectives and error budgets (the canonical reference is the Google SRE Book chapter on Service Level Objectives, with the operational follow-up in the SRE Workbook chapter on alerting on SLOs). The names matter less than the practice they enable.

• SLOs chosen from the user's perspective, with two or three per service rather than ten. More SLOs means none of them shape behaviour.
• Error budgets: the inverse of the SLO, expressing how much unreliability the team is willing to spend in a window. Used up early in the month means slow down on releases. Healthy means feature work keeps moving.
• Multi-window burn-rate alerting turns SLOs from dashboards into pages: short window catches catastrophic failures, long window catches slow drift. Without burn-rate alerting, SLOs are observation, not operation. (The pattern is documented in the SRE Workbook.)
• Reliability work has its own backlog, prioritised against features. Not a wishlist after every incident.
• Regular game days that exercise failure modes (region failover, dependency outage, traffic spike) before they happen for real
• Capacity planning informed by data, not by anxiety

In practice

The pattern that works: define two or three SLOs per service, expressed from the user's perspective. Compute the error budget weekly. When the budget is healthy, ship feature work. When the budget is burning fast, slow down and fix the cause. The conversation about which incidents matter and which can wait becomes possible because there is a shared number to point at. Reliability becomes a quantified property of the platform, not an opinion debated at every retrospective. Teams that adopt this discipline stop having the recurring "how reliable do we need to be?" argument and start having data-grounded trade-off discussions instead.

A scenario that ties it together

A platform was launching a new region. The build had gone well. Day 1 was clean. Two weeks in, latency started creeping up during peak hours. Alerts fired on raw thresholds, but no one could tell which ones to trust. Incident calls turned into long debugging sessions because three different teams owned overlapping pieces of the request path.

The team did not start by buying a new tool. They started by treating operations as engineering work. The dashboard was redesigned around the user journey. Alerts were audited and most were demoted or removed. Roles for incident response were written down. A short runbook covered the top failure modes. Releases were broken into canary slices behind feature flags.

None of this was new. It was discipline applied consistently to work that was previously assumed to be someone else's. The next region launch took half the effort, and the team's mean time to restore on the failures that did happen was measurably lower.

What teams get wrong

The common pattern is treating Day 2 as the cost of Day 1. Teams design beautifully, ship fast, then quietly absorb the operational debt. Dashboards proliferate. Alerts grow louder. Postmortems pile up.

The fix is not more dashboards. It is treating operations as engineering work with the same rigour as feature delivery. Operability is a property the system either has or does not. It is not earned by adding monitoring. It is earned by designing for visibility and operating with discipline.

Where to start

The most concrete starter from this post: an alert audit. List every alert that fires in the next week and apply a single test to each one: "what action would I take in the next five minutes?" Demote the alerts that have no answer. Remove the alerts where the answer is the same as another alert's. The audit takes a morning. The result usually halves alert volume and lifts trust on what remains, which is the precondition for every other operational practice in this post.

The shift

The most important shift in maturity is not technical. It is in stance.

The shift is from shipping software to operating systems:

• Operations is not a phase that follows engineering. It is engineering.
• Reliability is not a milestone reached. It is a discipline practiced.
• Incidents are not interruptions to the work. They are the work.

The teams that internalise this shift run platforms that are smaller, calmer, and more trusted. They do not have fewer incidents because their systems are more advanced. They have fewer incidents because their operational discipline is more consistent. Part 3 of this series argues that the same discipline applies again, in a different domain: the practices that make platforms operable are the practices that make AI useful in delivery.

---

Want to discuss? What is the one operational practice your team adopted that changed how you sleep at night? Drop a comment with patterns you have seen in your environment. Every reply gets read.

Previously in this series: Building Cloud Native Platforms That Scale: Patterns That Actually Work. The first post covered the design choices that make scale possible.

Next in this series: AI-First Platform Engineering: From Copilot to Agentic Delivery. Cloud helped us scale infrastructure. The next post looks at how AI is now changing how we build and run platforms.