Debugging a Thundering Herd at the Origin

Problem Statement

Origin request rate is flat at forty per second and then, once a minute, jumps to two thousand for about a second. Nothing about user behaviour is periodic at that interval. The spikes are the moment a popular object’s freshness lifetime elapses at every edge location simultaneously, and every one of them goes to the origin at once.

Prerequisite Concepts

Step-by-Step Resolution

Step 1 — Look for periodicity at the lifetime interval

The distinguishing feature of a herd is that it repeats at exactly the freshness interval:

# Requests per second at the origin, bucketed
awk '{print $4}' origin-access.log | cut -d: -f2-4 \
  | uniq -c | sort -rn | head -20

Spikes 60 seconds apart with an s-maxage=60 on the busiest object is not a coincidence; it is the mechanism.

The signature of a synchronised expiryBetween expiries the origin is nearly idle; at each lifetime boundary every location misses at once and the origin absorbs the whole footprint in a fraction of a second.Quiet40 req/s baseline0sExpiry2000 req/s from every PoP59s60s
One 60-second lifetime, seen from the origin

Step 2 — Confirm the requests share one key

A herd concentrates; a traffic spike spreads:

# Group the spike second's requests by URL
grep '14:22:07' origin-access.log | awk '{print $7}' | sort | uniq -c | sort -rn | head

One URL accounting for the overwhelming majority confirms the diagnosis. A flat distribution across many URLs is genuine traffic and needs a different response.

Three spikes that look alike in a request-rate graphPeriodicity and key concentration separate an expiry stampede from a purge repopulation and from real traffic growth.PeriodicConcentrated on one keyCorrelates withExpiry stampedeyes, at the lifetime✓nothing externalPurge repopulation✗✓a purge log entryGenuine traffic spike✗✗a campaign or referral
Distinguishing the three causes from the same graph

Step 3 — Enable request collapsing

Collapsing removes the concurrency within a single cache: the first request fetches and the rest wait for it.

# Varnish coalesces by default; the mistake is passing objects that could coalesce
sub vcl_recv {
    if (req.url ~ "^/api/stream") { return (pass); }   # genuinely per-request
    return (hash);                                     # everything else coalesces
}

Marking something pass that could have been shared is the most common reason collapsing appears not to work.

Step 4 — Add a shield tier

Collapsing bounds each location to one origin request. With two hundred locations that is still two hundred simultaneous requests. A shield reduces it to one, because every location’s miss is routed through the same upstream cache.

Origin requests in the second after expiryCollapsing bounds each location to one request; a shield reduces the number of locations that can miss independently; together they reduce the herd to a single fetch.Neither1840reqCollapsing only194req one per PoPShield only212reqBoth1reqsame object, same traffic, four configurations
Origin requests during one expiry event

Step 5 — Desynchronise the expiry

Even with both mechanisms, a purge or a deploy can re-synchronise every location. Adding a small random component to the lifetime spreads subsequent expiries:

sub vcl_backend_response {
    # Spread expiry over a 10% window so locations do not align
    set beresp.ttl = beresp.ttl * (0.95 + (std.random(0, 100) / 1000));
}

The stronger version of the same idea is a grace window, which removes the blocking refresh entirely: the edge answers from the expired copy and refreshes behind the request, so no client waits and the origin sees one refresh rather than a queue.

Cache-Control: public, s-maxage=60, stale-while-revalidate=30

Step 6 — Decide which mitigation to reach for first

The four mitigations are not interchangeable, and applying them in the wrong order wastes effort. A grace window is almost always the first thing to try: it is a single directive, it removes the user-visible latency immediately, and it works even when the herd is caused by something you have not identified yet. Collapsing is next, because it is usually already available and only needs to be confirmed rather than built. A shield is a larger change with a billing consequence and is worth reaching for when the footprint itself is the problem — many locations, a wide catalogue of rarely-requested objects, a single origin region. Jitter is last, because it treats the symptom rather than the cause and complicates reasoning about how long content is actually cached.

The one case that inverts this order is content that genuinely must never be served stale. There a grace window is not available, so collapsing and shielding carry the whole load and the lifetime has to be short enough that the herd, when it arrives, is something the origin can absorb.

Expected Output / Verification

After the mitigations, the origin’s request rate should be smooth rather than periodic, and the busiest object should account for a proportion of origin traffic close to its share of distinct content rather than its share of demand:

before:  baseline 40 req/s, peaks 2000 req/s every 60s
after:   baseline 41 req/s, peaks 55 req/s, no periodicity

Verify with a synthetic burst as well: issue twenty concurrent requests for a single expired key and confirm the origin logs one fetch, not twenty.

Edge Cases

  • An object marked pass unnecessarily. Coalescing cannot apply to something the cache was told not to store, so a stray pass reintroduces the herd for that key alone.
  • Streaming or long-polling responses. These genuinely cannot coalesce, and forcing them to will serialise clients behind one another. Exclude them explicitly.
  • A purge re-synchronising everything. A bulk purge sets every location’s expiry to the same instant, recreating the herd on the next lifetime boundary. Jitter applied at storage time survives this; jitter applied only at first fetch does not.
  • A shield in the wrong place. A shield close to users rather than close to the origin reduces neither the number of locations nor the round-trip cost. Placement is an origin-proximity decision.
  • Collapsing hiding a slow origin. With a queue behind one fetch, a slow origin response becomes a slow response for every queued client at once. Monitor queue depth alongside request rate.
  • Health checks synchronised with expiry. A monitor requesting the same object at a fixed interval can itself be the trigger, arriving just after expiry every time and warming the cache with a single-request pattern.

Frequently Asked Questions

How is a thundering herd different from a traffic spike?

A traffic spike raises requests across many URLs; a herd concentrates on one key and repeats at exactly the freshness interval. Grouping the spike’s requests by URL separates the two in seconds.

Does request collapsing alone solve it?

It solves the concurrency within a single cache. With many independent edge locations you still get one origin request per location at the same instant, which is what a shield tier reduces.

Why does adding jitter help?

Because the herd exists only because expiry is synchronised. Spreading lifetimes over a small random range means locations expire at different moments and the origin sees a smooth trickle instead of a wall.

Is a grace window a valid mitigation?

Yes, and often the most effective one. With stale-while-revalidate the edge answers instantly from the expired copy and refreshes behind the request, so the origin sees one refresh rather than a queue of blocked clients.

Back to Origin Shielding and Request Collapsing