AI KV Cache Memory Tier Calculator

What Is KV Cache in AI Inference?

KV cache is the memory that stores attention keys and values for tokens already processed by a transformer model. During generation, the model can reuse that state instead of recomputing attention across the full prompt and generated history. That reuse is one reason modern inference servers can serve long context windows, multiple users and streaming token output at useful speeds.

The difficult part is scale. A single short prompt may not create a large memory problem, but thousands of concurrent sequences with long active contexts can create many terabytes of cache pressure. Once the working set outgrows local accelerator HBM, the system must choose between reducing concurrency, shortening context, adding more accelerators, adding a shared memory tier or falling back to storage.

How Much Memory Does KV Cache Use?

The calculator uses a transparent planning formula: concurrent sequences times context tokens times KV bytes per token times active context utilization. The bytes-per-token input is intentionally editable because the true value depends on the model architecture, layer count, hidden size, attention layout, quantization, runtime and cache format. Model-size labels are shown for planning context only; they do not silently change the formula.

Prompt cache reuse is shown separately. A high reuse rate can reduce incremental KV cache growth when many users share prefixes or system prompts, but it does not make raw memory pressure disappear. Seeing both raw and effective cache prevents teams from building a plan that only works when reuse is perfect.

Why KV Cache Outgrows HBM

HBM is the fastest and closest memory tier for accelerators, but a large share is usually consumed by model weights, runtime state, activation workspace and framework overhead. The calculator therefore starts with total HBM, subtracts the reserved percentage and models only the remaining capacity as available for KV cache. If the effective cache exceeds that amount, overflow becomes a memory-tier design question rather than a simple GPU count question.

Adding accelerators increases HBM capacity, but it can also increase total serving demand. The practical goal is not simply to maximize memory; it is to keep the hottest cache close to compute, place warm cache in a lower-cost shared memory tier and avoid letting too much of the active working set become storage-bound.

HBM vs DDR5 vs NVMe for KV Cache

What Is a Shared AI Memory Tier?

A shared AI memory tier gives several accelerators access to memory outside their local HBM. It can be implemented with CXL memory expansion, DDR5 memory appliances, optical links or other rack-scale fabrics. The goal is not to make DDR5 behave exactly like HBM; it is to provide a larger intermediate tier so warm KV cache does not immediately fall to persistent storage.

This is where architectures such as Marvell Photonic Fabric Memory become relevant. The announced approach combines DDR5-backed pooled memory, local HBM caching and optical connectivity. The calculator remains vendor-neutral because the sizing problem exists whether the shared tier is optical, electrical, CXL-based or built into a future platform design.

How CXL Memory Pooling Works

CXL is a coherency and interconnect protocol that can expose memory expansion resources to hosts and accelerators. In an AI inference design, CXL may help attach external DDR5 capacity without treating it like a block device. That matters because memory semantics are very different from reading and writing files or object chunks from storage.

CXL does not remove the need to model capacity and bandwidth. A system can have enough pooled DDR5 capacity and still bottleneck if the fabric cannot move cache data at the rate required by the active sequences. This calculator therefore checks both configured shared-memory capacity and configured shared-memory bandwidth.

Why Optical Memory Fabrics Matter

Electrical connectivity becomes harder as distances, lane counts and bandwidth demands grow. Optical fabrics are attractive for rack-scale or pod-scale memory sharing because they can target longer reach and high aggregate bandwidth with different power and cabling tradeoffs. Vendor claims should still be treated as implementation-specific until production systems are benchmarked under real workloads.

The practical planning question is simple: if optical fabric makes a larger DDR5 pool reachable by more accelerators, how much HBM can remain dedicated to the hottest cache and how much lower-cost memory can absorb the warm working set? The answer depends on context length, concurrency, cache reuse and data movement per generated token.

How to Size DDR5 for AI Inference

DDR5 sizing starts after HBM allocation. The calculator places the HBM overflow into shared DDR5, then compares that requirement against configured DDR5 capacity. It also calculates approximate DIMM count across 64GB, 96GB, 128GB and 256GB RDIMMs. Whole DIMMs are required, so the final installed capacity is rounded up and unused capacity is shown explicitly.

Live median $/GB is used where enough current DDR5 RDIMM products exist for a capacity. If the market has insufficient data for a DIMM size, the calculator says so rather than inventing a price. That is important because larger DIMMs may improve slot efficiency while still carrying a different cost per GB.

When KV Cache Falls Back to NVMe

NVMe fallback is the remaining active KV-cache requirement after HBM and shared DDR5 are allocated. Enterprise SSDs can provide large capacity at a much lower cost per TB than HBM, but they are still persistent storage devices rather than memory. A fallback tier can be useful for cold or less latency-sensitive state; it is risky when a large share of the active cache must be read repeatedly during token generation.

The enterprise SSD optimizer compares 3.84TB, 7.68TB, 15.36TB, 30.72TB and 61.44TB capacities. It uses whole-drive counts, installed capacity and excess capacity so a planner can see the acquisition implication of each capacity class.

How Memory Bandwidth Affects Token Throughput

Capacity is not sufficient by itself. A tier can be large enough and still too slow if aggregate demand exceeds the available bandwidth. The bandwidth model estimates data movement from active sequences, tokens per second and bytes moved per generated token, then divides demand across HBM, shared memory and NVMe according to the expected hit-rate assumptions.

The output is intentionally an estimate rather than a benchmark. It helps identify whether a design is likely to be HBM-capacity bound, shared-memory-capacity bound, shared-memory-bandwidth bound, NVMe-capacity bound, NVMe-bandwidth bound or reasonably balanced under the assumptions entered.

How to Reduce KV Cache Infrastructure Cost

Cost reduction usually starts with workload discipline. Shorter active context, better prompt reuse, batching policies, context eviction, cache quantization and routing similar requests together can reduce the amount of new KV cache created. Hardware planning then decides whether more HBM, more DDR5, more shared-memory bandwidth or more NVMe capacity is the better use of budget.

For capital planning, compare this tool with the AI memory CAPEX calculator. That calculator models fleet-level server RAM and enterprise SSD CAPEX, while this one focuses on the memory hierarchy behind active inference.

KV Cache Formula Table

Frequently Asked Questions