Sizing a Varnish or Nginx Cache Store

Problem Statement

A proxy cache is configured with whatever size seemed reasonable when it was deployed, and hit ratio has been drifting downward as the catalogue grows. The question is not “how big should a cache be” in the abstract but “how much of the demand is this store failing to hold”, and that is measurable.

Prerequisite Concepts

Step-by-Step Resolution

Step 1 — Pick the measurement window

The store only needs to hold an object for as long as that object remains reusable. The correct window is therefore the longest freshness lifetime the store is expected to honour — if your longest s-maxage is 24 hours, measure 24 hours. Measuring a week over-sizes the store by counting objects that would have expired anyway.

Step 2 — Compute the working set

Count each distinct URL once and sum the response sizes:

# Access log fields: ... "$request" $status $body_bytes_sent ...
awk '$9 == 200 {size[$7] = $10} END {
  for (u in size) { total += size[u]; n++ }
  printf "%d distinct objects, %.1f GB working set\n", n, total/1073741824
}' /var/log/nginx/access.log

Two refinements matter. Exclude responses that are never stored — anything marked no-store, and any status the cache does not keep — because they consume no capacity. And count each variant separately if Vary is in play, since each is a full stored copy.

From access log to store sizeThe working set is the distinct bytes requested within one lifetime, and the store size is that figure plus headroom for growth.Access logone lifetime windowfilterDistinct URLscount each oncesum sizesWorking set90 GBadd headroomStore size130 GB
Four steps from log data to a configured ceiling

Step 3 — Configure the store

For Nginx the ceiling is max_size, and inactive is a second, independent eviction rule worth setting deliberately:

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=main:96m
                 max_size=130g inactive=24h use_temp_path=off;

The keys_zone size is separate from the content size and is often the forgotten constraint: it holds metadata, roughly one kilobyte per stored object, so 96 MB accommodates around a hundred thousand entries. A zone too small for the object count evicts entries regardless of how much disk remains.

For Varnish, storage is declared at startup and media is worth separating:

varnishd -s main=file,/var/lib/varnish/main.bin,130G \
         -s media=file,/var/lib/varnish/media.bin,400G

Step 4 — Verify with the eviction counters

Configuration is a hypothesis; the counters test it. Varnish exposes evictions directly:

varnishstat -1 -f MAIN.n_lru_nuked -f MAIN.n_object -f MAIN.cache_hit -f MAIN.cache_miss

A steadily rising n_lru_nuked means the store is full and every admission costs an eviction. After a correct resize it should fall to near zero and stay there.

For Nginx there is no eviction counter, so measure the directory against the ceiling:

watch -n 60 'du -sh /var/cache/nginx'

A directory pinned at max_size is a store in continuous eviction.

Eviction rate before and after sizing to the working setA store below the working set evicts continuously; once capacity exceeds demand within one lifetime the eviction rate collapses and hit ratio stabilises.40 GB store1840/min continuous eviction90 GB store210/min at the boundary130 GB store3/min expiry onlysame traffic, same content, three ceilings
Objects evicted per minute, before and after resizing

Expected Output / Verification

Three signals together confirm the store is no longer the constraint: eviction count near zero, the store size settling below its ceiling rather than pinned at it, and hit ratio that no longer falls as the catalogue grows. If hit ratio stays low while all three hold, the problem is elsewhere — most often key fragmentation or a directive preventing storage.

The three ceilings a store can hitDisk size is only one of the limits that evicts entries; the metadata zone and the inactivity timer each remove objects independently of how much disk remains.CHECKED CONTINUOUSLY1Disk sizemax_size on the zone2Metadata zonekeys_zone, ~1 KB per object3Inactivity timerinactive, regardless of freshnessEASIEST TO OVERLOOK
Three independent limits, any one of which evicts
MAIN.n_lru_nuked    3     0.00 /s
MAIN.n_object   84120        .
MAIN.cache_hit  918400   96.3 %

Step 5 — Decide whether more capacity is the right lever at all

A resize is worth doing only when the store is genuinely the constraint, and there are two common cases where it is not.

The first is key fragmentation. If a single page is stored under hundreds of keys because campaign parameters or an unsorted query string reach the cache key, the working set measurement is inflated by content that does not exist. Sizing to it buys disk for duplicates. The measurement in step 2 makes this visible: a distinct-object count far larger than the number of URLs the site actually publishes is the signature, and the fix is normalization rather than capacity.

The second is a periodic sweep. A crawler or a synthetic monitor that walks every URL once per night evicts the entire hot set in favour of objects nobody will request again, and the store recovers over the following hours. Hit ratio plotted by hour shows this immediately as a nightly cliff. Admission control — Nginx’s proxy_cache_min_uses, or an equivalent policy — solves it by refusing to store single-request objects, and no amount of extra disk will.

Where neither applies and the eviction counters are high, capacity is the answer and the measurement gives the number.

Edge Cases

  • The metadata zone, not the disk, is the ceiling. An Nginx keys_zone sized for fewer objects than the working set contains evicts entries while the disk sits half empty. Size it at roughly one kilobyte per expected object.
  • inactive shorter than the lifetime. Nginx removes entries untouched for inactive regardless of freshness, silently discarding the rarely-requested remainder. Keep it at or above the longest lifetime you intend to honour.
  • A working set that grows discontinuously. A catalogue import or a new locale can double the distinct-object count overnight. Re-measure after any event that changes what is addressable, not on a fixed schedule.
  • Filesystem overhead on small objects. A store full of two-kilobyte responses wastes a significant fraction of its disk on block granularity. Measure the directory size, not the summed content size, when checking against the ceiling.

Frequently Asked Questions

Should the store hold the whole catalogue?

Almost never. The catalogue can be orders of magnitude larger than the set of objects anyone requests within a lifetime, and capacity spent on objects nobody asks for returns nothing. Size to the working set and let the rarely-requested remainder miss.

What is a reasonable headroom factor?

Enough to absorb growth and traffic-mix changes without another resize — commonly 30 to 50 percent above the measured working set. Beyond that the returns are negligible, because hit ratio flattens once the working set fits.

Does a bigger store fix a low hit ratio?

Only when the store is the constraint. If hit ratio is low because the cache key fragments, or because a crawler flushes the store nightly, capacity postpones the symptom without touching the cause.

Should large media share a store with everything else?

No. A handful of large objects can evict thousands of small ones under a single LRU policy. Where the cache supports multiple stores, give media its own with its own ceiling.

Back to Cache Storage, Eviction and Capacity