Preventing Stale Serving During an Origin Outage

Problem Statement

During a recent origin incident the site stayed up, which was mostly good news — except that a checkout page continued displaying prices from before a pricing update, and an inventory view continued showing stock that had already sold. The caching layer did exactly what it was designed to do. The task is to keep that behaviour everywhere it helps and remove it precisely where it does not.

Prerequisite Concepts

• must-revalidate and proxy-revalidate explained — the directive that withdraws a cache’s permission to serve stale.
• Serving stale content with stale-while-revalidate — the grace directives that widen the window this procedure closes.
• Cache hit, miss and bypass mechanics — the states a stored entry moves through as it expires.

Step-by-Step Resolution

Step 1 — Classify the routes

The question for each route is simple and answerable by the people who own the feature: if a user acted on a value five minutes out of date, what happens? For a documentation page nothing happens. For a price, a balance, a stock level or a permission check, something possibly irreversible happens.

Keep the strict list short. Every entry converts a caching decision into an availability dependency on the origin.

Step 2 — Add the directive to the strict routes

location /api/inventory/ {
    add_header Cache-Control "public, max-age=10, s-maxage=30, must-revalidate" always;
    etag on;
}

location /checkout/ {
    add_header Cache-Control "private, no-store" always;
}

The short lifetime matters as much as the directive. A long lifetime with must-revalidate gives a long window of unverified reuse followed by hard failure — the worst of both properties.

Step 3 — Find the mechanisms that override it

This is the step teams skip, and it is where the incident comes from. At least four independent mechanisms can authorise stale serving, and only one of them is the response header.

In Fastly, grace is set independently of the header:

sub vcl_backend_response {
    if (bereq.url ~ "^/api/inventory/") {
        set beresp.grace = 0s;
        set beresp.stale_if_error = 0s;
    }
}

In Nginx, proxy_cache_use_stale is the equivalent and frequently inherited from a server-level block:

location /api/inventory/ {
    # Do not fall back to a stored copy when the upstream fails
    proxy_cache_use_stale off;
    proxy_cache_background_update off;
}

Cloudflare’s Always Online serves a stored copy when the origin is unreachable, which is exactly the behaviour being forbidden. It is a zone-level toggle in most plans, so confirm its scope before assuming a per-route header controls it.

Step 4 — Rehearse the outage

An untested strict route is an assumption. Point a staging environment’s origin at an address that refuses connections and walk the routes:

# Warm the entries, then wait past the lifetime with the origin down
for path in /api/inventory/9472 /guides/setup; do
  printf '%-24s ' "$path"
  curl -s -o /dev/null -w '%{http_code}\n' "https://staging.example.com$path"
done
# Expected: 504 for the strict route, 200 for the resilient one

Both outcomes matter. A 200 on the strict route means something is overriding the directive. A 504 on the resilient route means strictness has leaked somewhere it was not intended.

Step 5 — Make the strict list visible and reviewable

A strict route is a decision with an operational cost, and decisions with operational costs need an owner and a review date. Keep the list somewhere the people who run the site will actually see it — a comment block in the CDN configuration is better than a wiki page nobody opens during an incident.

Two pieces of information belong with each entry. The first is why the route is strict, in one sentence, phrased as the harm being avoided rather than as a policy: “a stale stock figure lets a customer buy something already sold” is reviewable, “correctness-critical” is not. The second is what the route does instead of serving stale — an error page, a degraded view, a client-side retry — because that behaviour is what users will actually experience, and it is usually worth designing rather than inheriting.

Reviewing the list quarterly tends to shorten it. Routes are marked strict during incidents, when caution is cheap and nobody is weighing availability, and the justification often stops applying once the underlying data model changes.

Expected Output / Verification

The rehearsal produces a two-column result that matches the classification exactly: strict routes error, resilient routes serve. Anything that disagrees points at a mechanism you have not yet found.

/api/inventory/9472      504
/checkout/               502
/guides/setup            200
/assets/app.b7f31e.js    200

Edge Cases

• Grace inherited from a service-wide default. A VCL grace value set once at the service level applies to every backend response unless overridden per route, and it silently outranks the header.
• A shield holding a copy the edge no longer has. Strictness must be applied at every tier. An edge that refuses to serve stale but forwards to a shield that does not produces the stale response anyway.
• must-revalidate on a route with no validator. Every request past expiry becomes a full refetch, so the origin is hit harder during recovery precisely when it is least able to cope.
• Browser-stored copies outside your control. Neither directive reaches an entry a browser already holds under a previously published long lifetime. Short browser lifetimes on strict routes are the only protection.
• Health checks masking the outage. A CDN that considers the origin healthy will keep revalidating rather than falling back, so the rehearsal must actually break the connection rather than slow it.

Frequently Asked Questions

Why do caches serve stale content during an outage by default?

Because RFC 9111 permits it and it is usually the better outcome. A slightly out-of-date page is more useful than an error for most content, so implementations default to resilience and require an explicit directive to opt out.

Is stale-if-error the opposite of must-revalidate?

Effectively yes. stale-if-error widens the window in which an expired response may be served when the origin fails; must-revalidate removes it entirely. Sending both is contradictory.

Does disabling Always Online affect routes I did not mark strict?

It can, because the setting is usually zone-wide rather than per-route. Where the vendor supports scoping it to a path pattern, scope it; otherwise weigh the loss of resilience on the rest of the site against the strictness you need.

How do I test this without breaking production?

Reproduce it in staging by stopping the origin, or by pointing the CDN at an origin address that refuses connections. Never rehearse an outage by disabling a production origin during traffic.

Related

• How must-revalidate interacts with max-age covers the pairing that decides when the strict window begins.
• Implementing stale-while-revalidate on Cloudflare is the configuration you are deliberately not applying to strict routes.
• Cache-Control best practices for REST APIs gives the endpoint classification this procedure depends on.

Back to must-revalidate and proxy-revalidate