Request Collapsing Limits with Streaming Responses

Problem Statement

A server-sent events endpoint behaves perfectly with one client and degrades sharply with ten. The origin is not slow — its own timings are unchanged. The cache in front of it is coalescing requests for the same key, waiting for a response that by design never finishes, and every additional client joins a queue that will not drain.

Prerequisite Concepts

Step-by-Step Resolution

Step 1 — Identify the response types that cannot coalesce

Coalescing assumes that one upstream response can satisfy many clients. Three categories break that assumption:

Response types and whether coalescing is safeCoalescing is safe when one completed response can serve many clients, and harmful whenever a response is open-ended or unique per request.Safe to coalesceWhyCacheable document✓one body serves everyoneCacheable API read✓identical payload per requestServer-sent events✗never completesLong-polling endpoint✗completes only on an eventPer-user response✗body differs per clientUpload or mutation✗not a shared read at all
Three categories that must bypass coalescing

Step 2 — Recognise the queueing symptom

The signature is latency that scales with concurrency while origin latency does not:

# Ten concurrent requests to the same streaming route
for i in $(seq 1 10); do
  curl -s -o /dev/null -w "%{time_starttransfer}\n" \
    https://example.com/api/stream/notifications &
done | sort -n

Times spread from milliseconds to many seconds, increasing with position, means the requests are being serialised. Times all similar means they are being served independently, which is what you want.

What happens to the second clientThe first request creates a busy object; every later request for the same key waits for it to complete, which for a stream is never.ClientsCacheOriginclient A: GET /api/streamforwarded, response opensclients B..J: same keyqueued behind the busy objectstream stays open indefinitely
One stream, one busy object, a queue that never drains

Step 3 — Mark the routes as pass

The fix is to tell the cache these responses are not shareable, before it creates a busy object for them:

sub vcl_recv {
    # Open-ended and per-request responses must never coalesce
    if (req.url ~ "^/api/(stream|events|poll)/") { return (pass); }
    if (req.http.Accept ~ "text/event-stream")   { return (pass); }
    if (req.method != "GET" && req.method != "HEAD") { return (pass); }
    return (hash);
}

The Accept check catches clients negotiating an event stream on a URL that also serves an ordinary representation, which a path pattern alone would miss.

In Nginx the equivalent is to disable both caching and buffering for the location, since buffering alone will hold a stream until it completes:

location /api/stream/ {
    proxy_pass http://origin;
    proxy_cache off;
    proxy_buffering off;
    proxy_read_timeout 3600s;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
}

Step 4 — Confirm coalescing still works where it helps

Excluding too much is the opposite failure, and it reintroduces the stampede these mechanisms exist to prevent:

# Twenty concurrent requests for a shared, cacheable object
for i in $(seq 1 20); do
  curl -sI https://example.com/products/9472 | grep -i 'x-cache' &
done | sort | uniq -c

One miss and nineteen hits confirms coalescing is still active on shareable content. Twenty misses means the pass rule is matching more than intended.

Time to first byte by concurrency on a streaming routeWith coalescing active, latency grows with the number of concurrent clients because each waits for the one before it; with the route excluded, latency is flat.1 client, coalesced0.1s5 clients, coalesced9.4s10 clients, coalesced21s10 clients, pass0.1sorigin response time unchanged in all four cases
Time to first byte on an event-stream endpoint

Step 5 — Keep the exclusion list narrow and documented

A pass rule is a permanent opt-out of every caching benefit for the routes it matches, so the list deserves the same scrutiny as any other performance-relevant configuration. Two habits keep it from spreading.

The first is to match as precisely as the traffic allows. A rule matching ^/api/ because one endpoint under it streams excludes every other API route from caching and coalescing at the same time, which is a large cost for a small fix. Match the streaming path, or the negotiated content type, and leave the rest of the namespace cacheable.

The second is to record why each entry exists. A rule added during an incident tends to outlive the endpoint it was written for, and a comment naming the route and the reason makes it possible to remove later. Reviewing the list when routes are retired usually shortens it, because streaming endpoints are frequently replaced by ordinary polling ones without anyone revisiting the cache configuration.

It is also worth confirming that the excluded routes are not carrying long lifetimes in their own headers. A route marked pass at the edge but advertising max-age=300 will still be stored by browsers, which produces a stale stream on reconnect and is difficult to attribute.

Expected Output / Verification

After the exclusion, streaming latency is flat across concurrency and shared objects still collapse:

/api/stream/notifications   10 concurrent, TTFB 0.09-0.12s, 10 origin connections
/products/9472              20 concurrent, 1 MISS + 19 HIT, 1 origin fetch

Both halves matter. Flat streaming latency with twenty misses on the product page means the exclusion is too broad and the herd is back.

Edge Cases

  • Buffering rather than coalescing. Nginx will hold a streaming response until it completes even with caching disabled, unless buffering is also turned off. The symptom looks identical.
  • A route serving both a stream and a document. Content negotiation, not the path, decides. Match on Accept as well as on the URL.
  • Chunked responses that do complete. A long but finite chunked response can safely coalesce, and excluding it loses a real benefit. Distinguish open-ended from merely slow.
  • Timeouts masking the problem. A read timeout will eventually release queued clients with an error, which turns an obvious hang into an intermittent failure that is much harder to attribute.
  • WebSocket upgrades. These are not HTTP responses in the cacheable sense at all and must bypass the cache entirely; a coalescing rule that catches the upgrade request breaks the connection outright.
  • Per-user responses without an identity dimension. If a per-user response shares a key with others, coalescing will serve one user’s body to everyone — a correctness failure rather than a latency one.

Frequently Asked Questions

Why does collapsing hurt a streaming response?

Because the cache holds later requests behind the first one, expecting to serve them all from the completed response. A stream never completes, so the queue never drains and every additional client waits indefinitely.

How do I recognise the symptom?

Latency on the affected route rises with concurrency while the origin’s own response time stays flat. The cache, not the origin, is the queue — which is why origin metrics look healthy throughout.

Does marking a route as pass disable caching for it?

Yes, and that is the intent. A response that is unique per request has nothing to share, so storing it wastes capacity and coalescing it actively harms latency.

What about server-sent events and long polling?

Both are open-ended by design and must be excluded for the same reason. Any response that stays open while more data arrives is incompatible with a mechanism that waits for completion.

Back to Origin Shielding and Request Collapsing