Building a Cache Hit Ratio Dashboard

Problem Statement

The CDN dashboard shows one number and it has fallen four points. That could be a cache key regression, a capacity limit, a purge, a crawler, a campaign, or nothing at all. A single figure cannot distinguish them, and the investigation that follows is usually slower than building the panels that would have answered it immediately.

Prerequisite Concepts

Cache debugging and observability — the headers these metrics are derived from.
Cache hit, miss and bypass mechanics — the outcomes being counted.
How to read and interpret the Age header — the signal that distinguishes a fresh hit from a revalidated one.

Step-by-Step Resolution

Step 1 — Collect the right fields

Everything downstream depends on four values per request: the URL, the cache-status token, the response size and the status code. Most CDNs can deliver these as a log stream; where they cannot, the origin’s own logs plus an edge status header are enough:

log_format cachelog '$time_iso8601 $request_uri $upstream_cache_status '
                    '$status $body_bytes_sent $request_time';
access_log /var/log/nginx/cache.log cachelog;

Step 2 — Chart request ratio and byte ratio side by side

They answer different questions and diverge more than most people expect:

One day of traffic, measured two ways

Step 3 — Add per-tier offload measured at the origin

Edge-reported hit ratio cannot see the shield, and it counts a revalidation as a hit even though a request reached the origin. Offload measured at the origin captures both:

Five panels, five distinct diagnoses

Step 4 — Break the ratio down by rank and by size

These two breakdowns catch the two most commonly misdiagnosed causes. A ratio that is high for the top thousand URLs and low for everything else is a capacity signature, not a configuration one. A ratio that falls monotonically with response size points at an object-size ceiling.

-- Hit ratio by request-count decile
SELECT decile,
       100.0 * SUM(CASE WHEN status = 'HIT' THEN 1 ELSE 0 END) / COUNT(*) AS hit_pct
FROM (SELECT *, NTILE(10) OVER (ORDER BY url_request_count DESC) AS decile FROM requests)
GROUP BY decile ORDER BY decile;

Step 5 — Annotate the events that legitimately move the number

Half of all hit-ratio investigations end at “we purged”. Overlaying purge calls, deploys and campaign starts on the timeline removes those cases before anyone opens a log:

Annotated dips need no investigation

Step 6 — Keep the panel count small enough to be read

A dashboard with thirty panels is consulted during an incident and understood by nobody. Five is close to the practical limit for something that has to be read under pressure, which means each panel has to earn its place by ruling out a specific cause.

The test for whether a panel belongs is whether it changes what you would do next. Request-versus-byte ratio does: it decides whether to investigate object count or object size. Origin offload does: it decides whether the problem is worth investigating at all, since a stable offload with a falling edge ratio is usually a purge rather than a defect. Ratio by rank does: it separates capacity from configuration. A panel showing hit ratio per PoP, by contrast, rarely changes anything — the variation between locations is mostly traffic-driven and it invites investigation of noise.

Everything else belongs in a query, not on the wall. Keeping the queries alongside the dashboard, named for the question they answer, gives the same coverage without diluting the five panels that need to be legible at a glance.

Expected Output / Verification

A working dashboard answers “why did it change” without a log query. The test is retrospective: take the last three hit-ratio incidents and check whether the panels would have identified each cause immediately. If any of them still requires an ad-hoc query, that query is the panel that is missing.

request hit ratio   96.2%  (-0.3 vs 7d)
byte hit ratio      61.4%  (-11.2 vs 7d)   <- the real change
origin offload      88.1%  (-0.4 vs 7d)
ratio, top decile   99.1%
ratio, bottom decile 42.0%  (-18 vs 7d)   <- capacity, not configuration

Edge Cases

Sampled logs. A one-percent sample is fine for the headline ratio and useless for the tail breakdown, because the tail is exactly what sampling discards.
Counting 304 as a hit. It saves bandwidth but not an origin request. Folding it into either category hides one of the two effects; chart it separately.
Mixed status vocabularies. Chained proxies produce compound tokens. Normalise them before aggregating or the counts will silently omit whatever does not match the expected string.
Ranged requests inflating counts. One video view can register as hundreds of block requests. Exclude or separate media before computing a site-wide ratio.
Time zones on annotations. Purge logs and CDN logs frequently disagree about zone, which makes an annotated dip appear an hour away from its cause.
Alerting on the headline number. It moves for legitimate reasons several times a week. Alert on sustained origin-offload decline instead, which is far quieter and far more meaningful.

Interpreting X-Cache and CF-Cache-Status headers covers the status tokens these panels aggregate.
Cache storage, eviction and capacity explains the capacity signature the rank breakdown reveals.
Measuring purge latency across CDN PoPs supplies the purge annotations the timeline needs.

Back to Cache Debugging and Observability