Using Vary Accept-Encoding Without Fragmenting Cache

Problem Statement

When an origin emits Vary: Accept-Encoding, every distinct encoding value in a client’s Accept-Encoding request header produces a separate entry in the CDN’s cache. A gzip client and a brotli client requesting the same URL each trigger a MISS, even though the payload differs only in compression format. The result is fragmented storage, a lower cache-hit ratio, and avoidable upstream fetches — all from a header that should help, not hurt.

Prerequisite Concepts

Before applying the fixes below, make sure you understand how the Vary response header drives cache key construction. Mapping Vary Headers to Edge Routing explains the three-phase mechanism (extract, normalize, variant lookup) that determines whether an existing cached object can satisfy an incoming request.

It also helps to understand how CDN Cache Keys Are Generated at the edge level — specifically how normalization rules vary across providers and why identical requests can still generate divergent keys.

Finally, review how public vs private cache scope interacts with Vary. A Cache-Control: private directive prevents CDN storage entirely, so fragmentation cannot occur — but the trade-off is zero edge caching.

Step-by-Step Resolution

Step 1 — Reproduce the fragmentation

Confirm that your origin is the source of the Vary: Accept-Encoding header and that the CDN is not already normalizing it away.

# First request — gzip client
curl -sI -H 'Accept-Encoding: gzip' https://example.com/api/data.json \
  | grep -iE 'x-cache|cf-cache-status|content-encoding|vary'

Expected (unfixed) output:

x-cache: MISS
content-encoding: gzip
vary: Accept-Encoding

# Second request — brotli client
curl -sI -H 'Accept-Encoding: br' https://example.com/api/data.json \
  | grep -iE 'x-cache|cf-cache-status|content-encoding|vary'

If this also returns x-cache: MISS, the CDN is creating a separate cache entry per encoding. Fragmentation is confirmed. If both requests return HIT after the first, the CDN is already normalizing and no further action is needed.

Step 2 — Enable CDN-level compression normalization

The cleanest fix is to instruct the CDN to normalize Accept-Encoding before it constructs the cache key. This collapses all encoding variants into a single stored object; the CDN then re-compresses on the way out to match each client’s declared capability.

Cloudflare — Enable Speed > Optimization > Content Encoding (previously labelled “Auto Minify” / “Brotli”). When active, Cloudflare normalizes Accept-Encoding to either br or gzip before the cache lookup, stores one object, and serves each client the encoding it supports. No origin change is required.

Fastly — Add a normalization snippet to vcl_recv:

if (req.http.Accept-Encoding ~ "br") {
    set req.http.Accept-Encoding = "br";
} elsif (req.http.Accept-Encoding ~ "gzip") {
    set req.http.Accept-Encoding = "gzip";
} else {
    unset req.http.Accept-Encoding;
}

This rewrites the header to a canonical value before the cache hash is computed, so gzip, deflate, br and br both resolve to the same cache key.

AWS CloudFront — In your CloudFront distribution’s Cache Policy, set “Compress objects automatically” to Yes and add Accept-Encoding to the allowlisted headers with normalization rather than the raw header list. CloudFront will then recognize gzip and brotli as equivalent for cache-key purposes and store a single object per URL.

Step 3 — Strip Vary at the origin or proxy layer

If CDN-level normalization is not available, a complementary approach is to prevent Vary: Accept-Encoding from reaching the cache at all. The CDN then stores a single unencoded (or pre-compressed) entry and handles delivery itself.

Nginx — strip Vary for static assets and offload compression to the edge:

location /assets/ {
    gzip_static on;
    proxy_hide_header Vary;
    add_header Cache-Control "public, max-age=31536000, immutable";
    proxy_pass http://backend;
}

gzip_static on serves pre-compressed .gz files when the client accepts gzip. proxy_hide_header Vary removes the upstream Vary: Accept-Encoding before the response is forwarded, so the CDN sees no Vary field and stores a single cache entry for the URL.

Apache — Remove Vary from specific paths using Header unset:

    Header unset Vary
    Header always set Cache-Control "public, max-age=31536000, immutable"

Combine with mod_deflate or mod_brotli at the Apache level if you want origin-side compression for non-CDN traffic paths.

Step 4 — Align Cache-Control to enable edge storage

Vary normalization only matters if the CDN is permitted to store the response at all. Verify that your origin emits a Cache-Control directive that allows shared caches to store the response:

Cache-Control: public, max-age=3600
Vary: Accept-Encoding

If the origin sends Cache-Control: private or Cache-Control: no-store, the CDN will not cache the response regardless of Vary. no-store prohibits storage entirely — max-age or s-maxage values alongside it are meaningless. Understand the full interaction in Mastering max-age and s-maxage Directives.

For APIs where you want CDN caching with per-encoding variants collapsed, the following combination works well with normalization enabled:

Cache-Control: public, s-maxage=300, stale-while-revalidate=60
Vary: Accept-Encoding

s-maxage sets the CDN TTL independently of the browser TTL. After normalization, the CDN stores one entry and serves it for 300 seconds, refreshing in the background during the stale-while-revalidate window.

Expected Output / Verification

After deploying the fix, run the full verification sequence:

# Populate the cache with a gzip request
curl -sI -H 'Accept-Encoding: gzip' https://example.com/api/data.json \
  | grep -iE 'x-cache|cf-cache-status|cache-status|vary|content-encoding|age'

# Immediately test a brotli client — should now hit the same entry
curl -sI -H 'Accept-Encoding: br' https://example.com/api/data.json \
  | grep -iE 'x-cache|cf-cache-status|cache-status|vary|content-encoding|age'

# Confirm both subsequent gzip requests also hit
curl -sI -H 'Accept-Encoding: gzip' https://example.com/api/data.json \
  | grep -iE 'x-cache|cf-cache-status|age'

Successful output after the first request:

• X-Cache: HIT (or CF-Cache-Status: HIT, or cache-status: "example"; hit) on both the brotli and the second gzip request.
• Age is a non-zero positive integer and increases on repeated requests, confirming the same stored entry is being reused.
• Vary is absent or set to a value that does not include Accept-Encoding (when the Nginx strip approach is used), or still present but only one cache entry is being served regardless.
• Content-Encoding matches the client’s preference — the CDN is re-encoding from a single stored object.

In Chrome DevTools: open the Network tab, filter by Fetch/XHR, and inspect the Size column. Entries showing (disk cache) or (memory cache) for different encoding contexts confirm consolidation. In the Response Headers pane, verify Vary is not listing Accept-Encoding when the strip approach is active, or that the CDN cache-status header reads HIT in both encoding scenarios when normalization is active.

If Age stays at 0 and you continue to see MISS after the fix, check that your CDN cache rule or Page Rule is actually matching the URL path and that the Cache-Control directive permits storage (no private, no-store, or no-cache without a validator present).

Edge Cases

• Legacy clients without brotli support — devices that omit br from Accept-Encoding must receive gzip fallback. CDN-level normalization handles this automatically: the CDN checks the header, recognises the absence of br, and serves gzip from the single stored entry. No origin change is needed.

• Dynamic API endpoints that genuinely vary by encoding — tightly compressed JSON streams (e.g., newline-delimited NDJSON) may have encoding-dependent semantics in some streaming contexts. In these cases, keep Vary: Accept-Encoding intentionally and mitigate origin load via origin shielding and request collapsing rather than eliminating the Vary field.

• HTTP/2 and HTTP/3 connections — ALPN negotiates the transport protocol (h2, h3); content encoding is still negotiated through Accept-Encoding exactly as in HTTP/1.1. CDN normalization logic applies regardless of transport. Verify your CDN’s normalization covers both h2 and h3 client connections — some older CDN configurations only normalize on HTTP/1.1 paths.

• Multi-CDN environments — When traffic is split across multiple CDN providers, each must apply identical normalization rules. Inconsistent normalization produces different hit rates across vendors and makes debugging cache behavior extremely difficult. Standardize normalization configuration across all providers before routing production traffic through multiple edges.

Related

• The no-cache vs no-store cluster explains when each directive is appropriate and how they interact with Vary on dynamic endpoints that should never be shared.
• How to Debug CDN Cache Key Mismatches covers the broader diagnostic workflow for any cache key divergence, including query string and cookie fragmentation.
• Tag-Based Cache Invalidation Patterns is the right tool for purging all encoding variants at once when you need to force a fresh fetch after a content change.

Back to Mapping Vary Headers to Edge Routing