Model GPU Memory and Concurrency Capacity Calculator¶

Use estimates first to filter out clearly infeasible combinations, then measure with the target engine. The calculator cannot precisely account for quantization metadata, temporary Workspace, CUDA Graph, fragmentation, MoE activated experts, or engine version differences.

1. Basic Formulas¶

Weight GiB ≈ parameter count × bytes per parameter / 2^30
Available GPU memory ≈ number of GPUs × per-GPU memory × (1 - runtime reserved ratio)
KV budget ≈ available GPU memory - weights - other resident overhead
Theoretical concurrency ≈ floor(KV budget / per-request KV Cache)

权重 GiB ≈ 参数量 × 每参数字节 / 2^30
可用显存 ≈ GPU 数 × 单卡显存 × (1 - 运行时预留比例)
KV 预算 ≈ 可用显存 - 权重 - 其他常驻开销
理论并发 ≈ floor(KV 预算 / 单请求 KV Cache)

BF16/FP16 weights use approximately 2 bytes per parameter, but total GPU memory is never equal to the weight size. Quantization also introduces Scale, Zero Point, packing formats, and Kernel Workspace. Training additionally requires gradients, optimizer states, and activations, so this inference calculator must not be used for training.

2. Per-Request KV Cache¶

KV Cache depends on the number of layers, KV heads, head dimension, precision, input length, generated length, and parallel sharding. The most reliable approach is to fix a request length in the target engine, read the engine metrics or GPU memory delta, and then enter it into the calculator.

3. Cost Calculator¶

The result covers GPU time only. Production cost should also include CPU, memory, storage, model distribution, network, idle redundancy, logging, control plane, and engineering support.

Further reading: Inference Optimization, Cost and Capacity