Why a 404 Response Gets Cached and How to Stop It

Problem Statement

A deploy briefly served 404 for a set of URLs. The deploy finished, the origin recovered within a minute, and the pages are still missing for a proportion of visitors an hour later. Nothing is wrong at the origin — a cache stored the error and assigned it a lifetime nobody chose.

Prerequisite Concepts

• Heuristic freshness and implicit caching — why a directive-free response gets a lifetime at all, and which status codes qualify.
• Cache hit, miss and bypass mechanics — the storability test that runs before any directive is read.
• Purging Cloudflare cache tags with the API — the mechanism you will need to clear a stored negative response.

Step-by-Step Resolution

Step 1 — Confirm the 404 is coming from cache

An error served from storage looks identical to a fresh one apart from Age:

curl -sI https://example.com/products/9472 \
  | grep -iE 'HTTP/|^age|cache-control|cf-cache-status|x-cache'

HTTP/2 404
Age: 2140
CF-Cache-Status: HIT

A HIT on a 404, with an Age of 35 minutes, is conclusive: the origin is not producing this response, storage is.

Step 2 — Determine what lifetime it was given

Check whether any directive was attached:

curl -sI https://example.com/products/9472 | grep -iE 'cache-control|expires|^date|^last-modified'

If Cache-Control is absent, the cache invented the lifetime. Error pages rarely carry Last-Modified, so most implementations fall back to a fixed default rather than a derived fraction — Varnish’s default_ttl of 120 seconds, or a CDN’s configured default, which is frequently much longer.

The 5xx rows are the reassuring ones: a server error cannot be stored by accident. The 4xx and 3xx rows are where incidents come from.

Step 3 — Clear the stored response

Purge the affected URLs so the next request repopulates from the recovered origin:

curl -sX POST "https://api.cloudflare.com/client/v4/zones/$ZONE_ID/purge_cache" \
  -H "Authorization: Bearer $CF_API_TOKEN" \
  -H "Content-Type: application/json" \
  --data '{"files":["https://example.com/products/9472"]}'

Then confirm the entry is gone:

curl -sI https://example.com/products/9472 | grep -iE 'HTTP/|^age|cf-cache-status'
# Expected: HTTP/2 200, Age: 0, CF-Cache-Status: MISS

Purging by tag is more reliable than by URL here, because a deploy fault usually affects a set of pages rather than one, and the URL list is rarely complete.

Step 4 — Make error responses explicitly uncacheable

The permanent fix is a directive on every error response. At the origin:

# Error responses must not outlive the condition that produced them
error_page 404 = @error404;
location @error404 {
    add_header Cache-Control "public, max-age=5" always;
    return 404;
}

error_page 500 502 503 504 = @error5xx;
location @error5xx {
    add_header Cache-Control "no-store" always;
    return 503;
}

The always flag is essential: without it, Nginx omits add_header on error responses, which is exactly the case being configured.

= 400">
    Header always set Cache-Control "public, max-age=5"

Step 5 — Add an edge guard

Even a correctly configured origin can lose a header in a rewrite. A rule at the edge that stamps a conservative directive on any error arriving without one closes the gap:

sub vcl_backend_response {
    if (beresp.status >= 400 && !beresp.http.Cache-Control) {
        set beresp.http.Cache-Control = "public, max-age=5";
        set beresp.ttl = 5s;
    }
}

Expected Output / Verification

After the change, an error response should carry an explicit directive and a short lifetime:

HTTP/2 404
Cache-Control: public, max-age=5
Age: 2
CF-Cache-Status: HIT

Five seconds of storage absorbs a crawler or a scan without any meaningful window of incorrectness. Verify by requesting a known-missing URL twice within five seconds — expect a hit — and again after ten, expecting a miss.

Edge Cases

• A 410 Gone that was not meant to be permanent. 410 is heuristically cacheable and semantically permanent. Using it for a temporarily removed resource produces exactly the outcome the status promises.
• A 301 cached in the browser. Browsers store permanent redirects aggressively and there is no purge path to a user’s own cache. A redirect published in error can persist for that user indefinitely; the only reliable mitigation is to publish 302 until you are certain.
• Error pages served by the CDN itself. A CDN-generated error may not pass through the origin’s header rules at all. Confirm what the edge emits when the origin is unreachable, not only what the origin emits when it is up.
• A deploy that produces 404 for assets rather than pages. Fingerprinted asset URLs are cached for a year on success. A 404 stored against one is far more damaging than against a page, because nothing will re-request it under a new name.

Frequently Asked Questions

Why is 404 cacheable at all?

RFC 9110 lists it among the statuses cacheable by default, because a missing resource is usually stably missing and repeatedly asking the origin about it wastes capacity. The design assumes the absence is real, which a deploy fault violates.

Does a 404 with no Cache-Control get a long lifetime?

It can. With no directive the cache applies its heuristic or its default TTL, and since error pages rarely carry Last-Modified the result is usually a fixed default of minutes to an hour — long enough to turn a brief fault into a visible outage.

Should error responses use no-store or a short max-age?

A short max-age is usually the better trade. no-store retains nothing but leaves the origin fully exposed to repeated requests for genuinely missing URLs. A few seconds absorbs scans while keeping the incorrectness window negligible.

Do redirects have the same problem?

Worse, because browsers store 301 aggressively and no purge reaches them. Prefer 302 until a redirect is genuinely permanent.

Related

• When does a browser invalidate a cached resource? explains why a browser-stored redirect is so difficult to withdraw.
• Event-driven cache invalidation with webhooks covers automating the purge that a deploy fault requires.
• Cache-Control best practices for REST APIs applies the same reasoning to error responses from an API rather than a page.

Back to Heuristic Freshness and Implicit Caching